D3.4 Assessment of Existing and Tailored Climate Services using a range of User-Driven Evaluation Metrics

Deliverable D3.4 Assessment of Existing and Tailored Climate Services using a range of User-Driven Evaluation Metrics June 2025 This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293 Innovating Climate services through Integrating Scientific and local Knowledge Deliverable Title: Assessment of Existing and Tailored Climate Services using a range of UserDriven Evaluation Metrics Author(s): Calum Baugh (ECMWF) and Katherine Egan (ECMWF) Contributing Authors(s): Emie Klein Holkenborg (510), Daniele Castellana (510), Deborah Dotta Correa (IHE), Schalk Jan van Andel (IHE), Rebecca Emerton (ECMWF), Christel Prudhomme (ECMWF), Andrea Popp (SMHI), Ilias Pechlivanidis (SMHI), Micha Werner (IHE) Date June 2025 Suggested citation: Baugh, C.A., Egan, K., Klein Holkenborg, E., Castellana, D., Correa, D.D., van Andel, S.J., Emerton, R., Prudhomme, C., Popp, A., Pechlivanidis, I., Werner, M. 2025. Assessment of existing and tailored climate services using a range of user-driven metrics. I-CISK Deliverable 3.4 Availability: ☒PU: This report is public ☐CO: Confidential, only for members of the consortium (including the Commission Services) Document Revisions: Author Revision Date Calum Baugh, Katie Egan Version 1; first draft May 2025 Calum Baugh, Katie Egan Version 2 June 2025 This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293 D3.4 – User-driven Evaluation of the I-CISK Climate Services Executive Summary This study describes a user-driven evaluation of six climate services within five of the living laboratories (hereafter referred to as living labs) of the I-CISK project. Climate services in this report refers to forecasts of hydro-meteorological variables at either medium range (up to 15 days ahead) or seasonal (up to 6 months ahead) time scales. User-driven evaluation goes beyond conventional statistical evaluation, as it aims to assess the value of a climate service for a user’s decision-making. For example, it could inform users about a climate service’s accuracy in forecasting the conditions when a particular decision-making criterion is met. The results aim to inform users how they could integrate the climate service forecasts within their decisionmaking process. The six climate services that were evaluated in this study are: 1) medium range forecasts of cold waves in Lesotho, 2) seasonal forecasts of meteorological drought in Lesotho, 3) seasonal streamflow forecasts in Rijnland (The Netherlands) 4) seasonal precipitation forecasts in Andalucia (Spain), 5) seasonal streamflow forecasts in Emilia Romagna (Italy), and 6) seasonal streamflow forecasts in the Alazani-Iori River Basin (Georgia). The user-driven evaluation in each climate service was tailored to the relevant users’ requirements, but broadly each evaluation followed a similar framework. The first step of the framework was to acquire the users’ local knowledge and requirements, in particular what decisions they needed to make, when they needed to be made, what criteria were used to make these decisions and whether they have a record of past events where these decisions were made. For some living labs (Spain, Netherlands and Georgia), this information was obtained through collaborative decision timeline exercises conducted with users. The second part of the framework evaluated the climate services in each of the living labs over a historical period, by applying the decision-making thresholds to the forecast and comparing against the user supplied record of past events. The skill of the forecast was computed using contingency table-based metrics which summarise the balance between the number of past events that were either hit, missed or warned in vain. For climate services which were based on probabilistic forecasts, the skill was computed separately for different forecasted probability thresholds of meeting the user defined threshold. The third step of the framework was to communicate the evaluation results to users. Results were visualised in different ways depending on the users’ requirements. For example, heatmaps were used to show how skill varied for different forecast probability thresholds and different lead times. Bar charts were used to show the breakdown of the hits, misses and false alarms, which users found to be helpful when interpreting the skill scores. The forecasts were also plotted for some past events which had been highlighted by users as having major impacts. Users found this to be a very intuitive way to compare how the forecasts performed against their knowledge of what happened during the events. Results across the evaluated climate services identified how they could be integrated into users’ decisionmaking, for example, by identifying the forecasted threshold exceedance probability of the users’ decisionmaking criterion which gave the best trade-off between hits, misses and false alarms. Presenting the skill scores and the break down of hits, misses and false alarms allowed users to judge the value of the climate services. For example, they could see what proportion of the forecasted threshold exceedances were false alarms. In some living labs, such as Italy, these results demonstrated that the forecasts had quite limited value, highlighting that further developments to the climate service, such as local bias correction, should be conducted before integrating the service into users’ decision-making. 1 D3.4 – User-driven Evaluation of the I-CISK Climate Services Future work could focus on integrating the results of the user driven evaluation directly into the visualisation of the realtime climate service forecasts. For example, highlighting when the forecast is predicting conditions aligned with a user’s decision-making criterion. 2 D3.4 – User-driven Evaluation of the I-CISK Climate Services Table of Contents 1 Introduction................................................................................................................................................ 1 1.1 Methodology Overview...................................................................................................................... 2 1.1.1 Understanding Users’ Decision-Making Context.........................................................................3 1.1.2 Evaluation of Climate Service to Predict Impactful Events..........................................................4 1.1.3 Presentation of Evaluation Results to Users...............................................................................6 2 Lesotho Living Lab.......................................................................................................................................7 2.1 Summary............................................................................................................................................. 7 2.2 Introduction to the Lesotho Living Lab...............................................................................................8 2.3 Cold wave evaluation.......................................................................................................................... 9 2.3.1 Coldwave Methodology............................................................................................................10 2.3.2 Reanalysis and re-forecast data................................................................................................12 2.3.3 Coldwave Evaluation Results.....................................................................................................22 2.3.4 Discussion: cold wave evaluation in the Lesotho LL..................................................................33 2.4 Lesotho LL – Drought evaluation.......................................................................................................35 2.4.1 Lesotho Drought Methodology.................................................................................................36 2.4.2 Results.......................................................................................................................................38 2.4.3 Discussion: Drought evaluation in the Lesotho Living Lab........................................................39 3 The Rijnland Living Lab (Netherlands).......................................................................................................41 3.1 Summary........................................................................................................................................... 41 3.2 Introduction......................................................................................................................................41 3.3 Methodology.................................................................................................................................... 42 3.3.1 Climate service and decision-making........................................................................................42 3.3.2 Streamflow Evaluation..............................................................................................................44 3.4 Results.............................................................................................................................................. 45 3.5 Discussions & Conclusions................................................................................................................ 48 4 Andalucia Living Lab (Spain)......................................................................................................................49 4.1 Summary........................................................................................................................................... 49 4.2 Introduction......................................................................................................................................49 4.3 Methodology.................................................................................................................................... 53 4.3.1 Data...........................................................................................................................................53 4.4 Results.............................................................................................................................................. 60 4.4.1 Historical precipitation..............................................................................................................60 3 D3.4 – User-driven Evaluation of the I-CISK Climate Services 4.4.2 Forecast evaluation................................................................................................................... 64 4.5 Discussion and conclusions for the Spain Living Lab.........................................................................78 5 Emilia Romagna Living Lab (Italy)..............................................................................................................83 5.1 Summary........................................................................................................................................... 83 5.2 Introduction......................................................................................................................................83 5.3 Methodology.................................................................................................................................... 83 5.3.1 Observation Data......................................................................................................................83 5.3.2 Forecast Data............................................................................................................................86 5.4 Results.............................................................................................................................................. 87 5.4.1 Evaluation of the Reforecast Data.............................................................................................87 5.4.2 Evaluation against Warnings Derived from Model Reforecast Climatology..............................89 5.5 Discussion & Conclusions: Italy Living Lab.........................................................................................93 6 Alazani-Iori River Basin Living Lab (Georgia).............................................................................................95 6.1 Summary........................................................................................................................................... 95 6.2 Introduction......................................................................................................................................95 6.3 Methodology.................................................................................................................................... 96 6.3.1 Climate service and decision-making........................................................................................96 6.3.2 Forecast evaluation and user feedback...................................................................................100 6.4 Results............................................................................................................................................ 100 6.5 Discussions & Conclusions: Georgia Living Lab...............................................................................106 7 Discussion...............................................................................................................................................107 7.1 A revised user-centred evaluation framework................................................................................107 8 Conclusions............................................................................................................................................. 109 9 References.............................................................................................................................................. 110 4 D3.4 – User-driven Evaluation of the I-CISK Climate Services List of Figures Figure 1 I-CISK co-creation framework and Living Laboratory locations. (MS10 ‘A prototype framework on cocreating end-user centred climate services’)......................................................................................................1 Figure 2 General framework of the user-driven evaluation applied in this study...............................................3 Figure 3 Decision timeline produced by Spanish living lab colleagues from UCM and CREAF during a participatory workshop in the Spanish living lab for Dehesa farmers.................................................................4 Figure 4 Contingency table approach used in the evaluation of each living lab in this study.............................5 Figure 5 Lesotho LL location and country zones as defined by LMS. Drought (top picture, from https://www.anticipation-hub.org/download/file-3526 ) and cold waves (bottom left, event in September 2024) are both impactful hazards for Lesotho’s people.....................................................................................8 Figure 6 Exposure rankings for villages in Lesotho (from Grab and Linde 2013; left), compared to ERA5 Land grids >2700 m amsl (top right) or LMS highland/lowland definitions (bottom right). Exposure rankings range from values of 1 (negligible exposure) to 18 (high exposure) and are used in the EAP to define at risk population. Note how the EAP definition of the highlands in the top right does not incorporate vulnerable villages around Quithing and Qacha’s Nek (red boxes) when applied to ERA5 Land orography, whereas the LMS definition does..........................................................................................................................................14 Figure 7 Example of identifying a cold event from 2016 in ERA5 Land using daily maximum temperature (red, °C) and daily snow accumulation (light blue, cm). The red bar indicated when impacts were recorded in our historical events database, and the red shading is the length of the event as detected by a 2°C, 0.5 cm threshold applied to ERA5 Land. Note that snow accumulation in this instance starts the day before temperature drops below the 2°C EAP threshold (dashed red line). The dark blue line is the 2-day rolling minimum (dark blue) of snow accumulation and dictates whether the event will be detected given the ‘2-day consecutive’ criterion of the EAP......................................................................................................................17 Figure 8 Cold wave event identified in ERA5 Land (as Figure 7) compared to the same event identified in a reforecast issued on July 18. The red bar indicated when impacts were recorded in our historical events database, and the red shading is the length of the event as detected by a 2°C, 0.5 cm threshold applied to ERA5 Land (top) or the re-forecast (bottom). Dashed lines are the 2°C (red) and 0.5 cm thresholds (blue)....18 Figure 9 Illustration of identifying events in each ensemble member of a re-forecast (red shading) and two different methods of calculating the probability of a start date from the individual ensemble members. The start method takes the percentage of members that actually start an event on each day, whereas the stack method takes the number of members where an event is occurring (so long as it is the first time that probability has been reached). Member zero is the control run and is included in the ensemble for evaluation purposes........................................................................................................................................................... 20 Figure 10 Number of ‘true’ and control re-forecast events at each lead time. Note that the actual number of real events doesn’t change, but the number of times they occur in a re-forecast does...................................21 Figure 11 Frequency bias by month (left), daily lead time (centre), and grouped lead times (right) for the control member of ECMWF medium range re-forecasts..................................................................................24 Figure 12 Hit, miss and false alarm counts and threat scores for the case that start dates need to match exactly (left) or can be within +-1 day (right). Both are shown for daily and grouped lead times. The plot in the centre shows counts of how many days out ‘near misses’ were (if the forecast was within 3 days).........24 Figure 13 Frequency bias for 2°C maximum temperature threshold exceedances. This was calculated for all our re-forecasts by comparing the number of instances of temperature at or below threshold to the number of instances on equivalent dates in ERA5 Land.................................................................................................25 5 D3.4 – User-driven Evaluation of the I-CISK Climate Services Figure 14 Frequency bias for 0.5 cm snow accumulation. This was calculated for all our re-forecasts by comparing the number of instances of snow accumulation at or above threshold to the number of instances on equivalent dates in ERA5 Land.....................................................................................................................26 Figure 15 Frequency bias and threat score when the ‘quantile’ spatial sampling method is used to define cold wave events...................................................................................................................................................... 26 Figure 16 Frequency bias (top row) and threat score when start dates are required to match exactly (middle row) and when a ±1 day tolerance is allowed (bottom row) for ensemble re-forecasts. Results for the ‘stack’ method of defining event start probabilities is shown in the right column and for the ‘start’ method in the left column........................................................................................................................................................28 Figure 17 Counts of hits, misses and false alarms by lead time for selected probability level for both the ‘stack’ and ‘start’ methods of defining event start from ensemble re-forecasts..............................................30 Figure 18 Schematic illustrating why false alarms and hits might both increase with lead time at higher probability levels in an overall bias ensemble forecast....................................................................................31 Figure 19 An example ROC curve for the ensemble re-forecast evaluation carried out here. The black dashed line is the ‘no skill’ line, and forecast curves that fall the left of it are considered skilful. This ROC curve would imply good skill for these forecasts,.................................................................................................................32 Figure 20 An example of the bar graphs that we showed to LMS and LRCS of ensemble probability vs. threat score, grouped by lead time. The gold colours show the control scores for comparison. These graphs were not considered easy to understand by users....................................................................................................33 Figure 21 Observed drought years across different datasets for the rainy season in Lesotho. The heatmap displays drought occurrence by year (x-axis) and observational data source (y-axis), with red cells indicating years classified as drought and light grey cells.................................................................................................36 Figure 22 Threat Score by forecast product and lead time for the DJF period across Lesotho. Bars represent the skill of each product in detecting events while accounting for missed events and false alarms. Values are derived from the results presented in Table 9 Forecast skill metrics for all products considered, for the DJF period across Lesotho. Observational data described in Section 2.3.1 is used as the reference ("truth"). Note that not all forecast products are available at all lead times: the EAP forecast begins in September, and the IRI forecast in August........................................................................................................................................ 38 Figure 23 Location of the Rijnland Living Lab in the Netherlands.....................................................................42 Figure 24 Rijnland Climate Service - Streamflow forecasts from i-cisk.dev.52north platform..........................42 Figure 25 Decision-making processes and drought thresholds in the Rijnland LL.............................................43 Figure 26 Average minimum discharge from observed values compared with monthly thresholds................43 Figure 27 Co-evaluation framework and criteria (Correa, 2024)......................................................................44 Figure 28 Sample size (number of pairs) by lead time and thresholds from April to September......................45 Figure 29 Visual inspection of forecasts from 2003 drought event for months of interest 1-7 months lead time. Forecast initiation time is in the title of each sub-plot............................................................................45 Figure 30 User feedback on event-based verification of forecasts...................................................................46 Figure 31 Rank histogram per lead time of interest.........................................................................................46 Figure 32 Mean absolute error of the ensemble average for different thresholds and user perception..........47 Figure 33 User perception on ROC curves and contingency tables...................................................................48 Figure 34 Location and elevation of the Guadalquivir River basin and Los Pedroches region, with annual average precipitation observations from AEMET (top) and land use categories in Los Pedroches (bottom, Masih et al., 2022 citing Broekman et al 2022).................................................................................................50 Figure 35 User feedback on conditions of precipitation and temperature that are ideal for each type of agriculture (green). Upper (temperature) and lower (precipitation) limits are coloured orange and included where provided. Lower temperature limits are in blue....................................................................................52 6 D3.4 – User-driven Evaluation of the I-CISK Climate Services Figure 36 Annual average rainfall 1993-2022 for sites that are within a 50 km buffer of the Los Pedroches region. Sites 89 and 465 are highlighted as they are considered in the text and their rainfall records are shown in Figure 45 and Figure 46.....................................................................................................................54 Figure 37 Comparison of observed and forecast timeseries for Los Pedroches average precipitation Seasonally (top), Annually (bottom left) and in October (bottom right)...........................................................55 Figure 38 Ensemble forecast means for rainfall in spring compared to eventual observed values in 2013 and 2014. Ideal values and adequate ranges are also provided. Note the effect of bias correction (bottom) is to increase mean rainfall relative to the raw SEAS5.............................................................................................57 Figure 39 Example forecast probabilities of ‘adequate’, ‘less than adequate’ and ‘more than ‘adequate’ rainfall for Spring 2013 and 2014 from SEAS5 and SMHI bias corrected SEAS5. The equivalent observed precipitation in each year is shown for comparison. These are the probabilities for the same forecasts as shown in Figure 38, but note that that figure shows the ensemble mean, whilst this shows the probability of being in each of our three categories...............................................................................................................59 Figure 40 Hydrological year historical rainfall from interpolated observations 1993-2022. Dashed lines enclose the range of values that farmers provided as moderate to good in the timelines exercise................60 Figure 41 October historical rainfall from interpolated observations 1993-2022.............................................61 Figure 42 Autumn historical rainfall from interpolated observations 1993-2022.............................................61 Figure 43 Winter historical rainfall from interpolated observations 1993-2022...............................................62 Figure 44 Spring historical rainfall from interpolated observations 1993-2022................................................62 Figure 45 Daily accumulated and seasonal rainfall for site 89 (see Figure 36 for location)..............................63 Figure 46 Daily accumulated and seasonal rainfall for site 465 (see Figure 36 for location)............................63 Figure 47 Frequency bias (left) and Threat score (right) for each category of rainfall (‘less than adequate’; top, ‘adequate’; middle, and ‘more than adequate’; bottom) in October. In each plot, SEAS5 results are shown on the left and SMHI bias corrected on the right. The zone of ‘topping out’ i.e. forecasting a given category all of the time, is delineated using the thick, dark blue line. Maximum threat score for each category/forecast is highlighted using a red box..............................................................................................66 Figure 48 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Less than adequate’ rainfall in October......................................................................................................................................................... 67 Figure 49 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Adequate’ rainfall in October. ..........................................................................................................................................................................68 Figure 50 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. Scores for forecasting ‘More than adequate’ rainfall in October. Only SMHI evaluation is shown because SEAS5 forecasts no events.................................................................................69 Figure 51 Frequency bias (left) and Threat score (right) for each category of rainfall (‘less than adequate’; top, ‘adequate’; middle, and ‘more than adequate’; bottom) in Autumn. In each plot, SEAS5 results are shown on the left and SMHI bias corrected on the right. The zone of ‘topping out’ i.e. forecasting a given category all of the time, is delineated using the thick, dark blue line. Maximum threat score for each category/forecast is highlighted using a red box..............................................................................................70 Figure 52 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Less than adequate’ rainfall in Autumn......................................................................................................................................................... 71 Figure 53 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Adequate’ rainfall in Autumn. ..........................................................................................................................................................................72 7 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 2 General framework of the user-driven evaluation applied in this study. 1.1.1 Understanding Users’ Decision-Making Context The first step in the framework involves understanding users’ decision-making context and local knowledge. There are many ways this could be conducted, but here we use a combination of timelines exercises (described in more detail in Van den Homburg et al., 2024, an example is given below in Figure 3 for the Spanish Living lab), documentation provided by users (EAPs) and direct discussion/collaboration. The following information was requested from each user: What decisions need to be made and when are these made? What climate service forecast(s) are used to make these decisions? What thresholds are applied to the climate service forecast(s) in order to make decisions? Do users have observations of past impactful events? For each decision identified by the users, they were asked to identify the climate services that they currently use to inform their decision. However, the stakeholders expressed a strong interest in the evaluation of the forecasts developed during the I-CISK project, therefore we focussed on those. Finally, stakeholders were asked what thresholds they used within their decision-making. These thresholds would be applied to the climate service to evaluate its value in identifying whether the forecasted value would exceed or not exceed the threshold value. 3 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 3 Decision timeline produced by Spanish living lab colleagues from UCM and CREAF during a participatory workshop in the Spanish living lab for Dehesa farmers. 1.1.2 Evaluation of Climate Service to Predict Impactful Events The second step of the framework evaluates the value of each climate service (that is the combination of the forecast and the decision thresholds identified in step 1) in predicting the impactful events identified by users. For each climate service evaluated in this work, the threshold associated with a particular decision was applied to all forecasts within the evaluation period to generate binary exceedance / non-exceedance (yes/no) predictions. These were evaluated against a time series of observed events, either provided by the stakeholder or derived from in-situ observations or re-analysis data. The evaluation of the forecasted and observed time series was done using a contingency table approach (Figure 4), this was chosen because of users’ feedback that the results from this approach were very intuitive. The contingency table summarises the number of correctly forecasted events (hits), false positives (false alarms), false negatives (misses) and correct negatives in a table (Figure 4): Hit: This category represents cases where an event was both forecasted and occurred, indicating a successful prediction. Miss: A miss occurs when an event happened, but the forecast did not predict it, highlighting a failure to anticipate the event. False Alarm: A false alarm happens when an event was forecasted but did not occur, which indicates an over-prediction of drought events. Correct Negative: This category represents situations where neither an event was forecasted nor occurred, reflecting a correct prediction of no event. The total values in each cell of the contingency table can be used to compute skill metrics (Table 1), to give an estimate of the forecast’s value in identifying events. The evaluation can be performed separately for each lead time within the forecast range, which can inform the user up to what lead time the forecasts can give valuable predictions. Ensemble forecasts were evaluated in some of the living labs, in these cases 4 D3.4 User-driven Evaluation of the I-CISK Climate Services different exceedance probabilities of the user defined threshold value were tested. More detail about the evaluation of ensemble forecasts is given in section 4.3.1.3 for example. Figure 4 Contingency table approach used in the evaluation of each living lab in this study. Table 1 Different skill metrics that were computed in this study to evaluate the value of the climate services. Note, not all metrics were computed in every living lab. Metric (perfect score) Formula Description Hit rate (Probability of Detection) (1.0) Hits / (Hits + Misses) Fraction of yes events correctly predicted False Alarm Ratio (0.0) False Alarms / (False Alarms+Hits) Fraction of predicted yes events which did not occur False Alarm Rate (Probability of False Detection) (0.0) False Alarms / (False Alarms + Correct Rejections) Fraction of observed non-events which were incorrectly predicted as yes events Threat Score (Critical Success Index) (1.0) Hits / (Hits+False Alarms+Misses) Fraction of observed and forecasted yes events which were correctly predicted Bias (1.0) (Hits + False Alarms) / (Hits + Misses) Ratio of the forecasted event frequency against the observed event frequency Accuracy (1.0) (Hits+ Correct Negatives) / (Hits+Correct Negatives+ Misses + False Alarms) Measures the proportion of correct forecasts out of the total number of events 5 D3.4 User-driven Evaluation of the I-CISK Climate Services 1.1.3 Presentation of Evaluation Results to Users The third and fourth steps of the evaluation are the generation of ‘Useful information’ by producing, collating and presenting the results to the stakeholders in a way which the climate services’ value was understandable and useful for decision-making. This could take the form of visual plots which summarise the results of the contingency table-based analysis. Examples of plots used in this study include the Relative Operating Characteristics (ROC) curve, heatmaps of the skill metric at different forecast lead times, bar charts summarising the number of hits, misses and false alarms, and plots produced for specific past events. These plots can quickly convey the lead times and threshold exceedance probability thresholds at which the climate services have value. Alongside these plots, guidance is presented to users on how the climate services could be used for their decision-making. For example, up to what lead time and which forecasted threshold exceedance probability should the climate service be used to make a particular decision. Users could then integrate this guidance into their operational decision-making, but this aspect was not dealt with in this study. The rest of this report outlines the user driven evaluation that was performed for each of the six climate services in the five living labs. For each climate service, the report outlines the decisions that users needed to make, the climate service that was evaluated, and the results of the evaluation. 6 D3.4 User-driven Evaluation of the I-CISK Climate Services 2 Lesotho Living Lab 2.1 Summary User-centred evaluation of cold wave and drought forecasts is performed in support of Early Action Protocols (EAPs) for Disaster Risk Reduction (DRR) developed and used by the Lesotho Red Cross Society (LRCS) and the Lesotho Meteorological Service (LMS) Cold Waves Cold wave early action trigger thresholds, which use maximum temperature and snow, are tailored to best identify historical, impactful cold wave events This tailoring finds an action trigger threshold of ‘maximum temperature 2°C or less with a snow accumulation of 0.5 cm or more, sustained for at least 2 days in the Lesotho Highlands’ to be suitable for identifying historical events in ERA5 Land re-analysis data ECMWF medium range re-forecasts issued between April and October, 2004-2022, are evaluated against ERA5 Land reanalysis data to test the ability of the forecasts to predict the start date of cold wave events identified using the tailored action trigger threshold at between 1 and 14 days lead time The evaluation supports the current EAP lead-time of 4-days for taking action, but cannot justify extending it a longer period ECMWF medium-range forecasts are found to be prone to false alarms, even at short lead times. In the best-case scenario (medium-range ensemble at 2-days lead time) approximately 30-40% of forecast cold wave events were found to be false alarms The methods presented demonstrate how user-centred evaluation employing a historical impacts database can overcome challenges associated with a lack of in-situ meteorological observations, presents a new event-based approach to contingency table verification, and considers the challenges of presenting evaluation results in a way that is both appropriate to the user-need and easy to understand. Drought Three sources of seasonal ensemble precipitation forecast (ECMWF’s SEAS5, SEAS5 downscaled by SMHI, and an IRI multi-model forecast) are evaluated against records of past drought events to identify the most suitable forecast provision to support long-lead time drought awareness at LRCS Forecasts issued in July, August and September for the December, January, February period are analysed, with drought defined using the EAP definition of a 25% probability of precipitation falling into the lower tercile of the climatological distribution. Whilst none of the three forecast streams exhibit strong skill (typically half or less of forecast drought events materialise), SEAS5 was found to be most suitable for use at LRCS based on both performance and data access/sustainability considerations The method highlights how user-centred evaluation can be used to select the most suitable forecast for a specific need, and demonstrates how considerations beyond forecast skill also need to feed into the process 7 D3.4 User-driven Evaluation of the I-CISK Climate Services 2.2 Introduction to the Lesotho Living Lab Lesotho is a mountainous country in southern Africa whose population is vulnerable to droughts and cold waves (Figure 5). The Lesotho Red Cross Society (LRCS) and Lesotho Meteorological Service (LMS) are the main stakeholders in the LL (Masih et al., 2022) and have collaborated to develop Early Action Protocols (EAPs) for both hazards. EAPs are commonly used for disaster risk reduction and facilitate ‘Early Warning, Early Action’ by outlining hazard prediction and response protocols. These protocols support timely mitigation action and resource allocation in the lead-up to a potential event (Harrowsmith et al., 2020). Figure 5 Lesotho LL location and country zones as defined by LMS. Drought (top picture, from https://www.anticipationhub.org/download/file-3526 ) and cold waves (bottom left, event in September 2024) are both impactful hazards for Lesotho’s people. The drought and cold wave EAPs for Lesotho provide a valuable starting point for user-centered evaluation because they contain ‘action trigger’ thresholds that specify the forecast values and lead times at which predetermined actions should be taken. Here, we analyze how well events identified by the thresholds are aligned with observations of past impactful events and then evaluate the ability of various forecasts to predict threshold exceedances. A contingency table approach is used because this is employed in the EAPs and is familiar to LMS and LRCS. The evaluation has two main aims: Provide information to support forecasting, preparedness, and future updates of the EAPs. Develop and test user-centred evaluation methods. Cold wave and drought analyses are presented separately below. 8 D3.4 User-driven Evaluation of the I-CISK Climate Services 2.3 Cold wave evaluation Cold Waves are defined by the International Federation of the Red Cross as ‘weather event[s] involving a cooling of the air, or the invasion of very cold air, over a large area … [that] can be preceded or accompanied by significant winter weather events, such as blizzards or ice storms’ (https://www.ifrc.org/our-work/disasters-climate-and-crises/what-disaster/cold-waves). In Lesotho and neighbouring South Africa, cold waves and associated snowfall have caused fatalities, damage to infrastructure, loss of livelihood, and substantial transport disruption, which have at times necessitated rescues operations (Appendix 1). As a result, LRCS and LMS have developed an EAP for cold waves, the only one of its kind in Africa. The ‘action trigger’ thresholds for this EAP are shown in Text Box 1 and combine two parameters (temperature and snow) with a time dimension (2 or 3 consecutive days below threshold to define an event). Actions are to be taken at a lead time of around 4 days. Text Box 1: Cold wave EAP action trigger thresholds taken from the March 2023 draft version, which was the current version at the time our analysis started. Defining the action trigger thresholds was challenging for LMS due to a lack of historical, in-situ observations, especially for snow (pers. comm. LMS). This is also a common challenge for forecast evaluation. Here, we demonstrate the use of reanalysis data as a substitute for in-situ observations. Reanalysis generates a ‘best estimate’ of historical conditions from model and observation data. It is therefore prone to model bias, and so our approach is to tailor the EAP action trigger thresholds (Text Box 1) using ERA5 Land reanalysis data such that the detection of historical, impactful cold wave events is maximised. A database of past events is created for this purpose and using it ensures that the tailored thresholds are linked to reality despite the lack of in-situ observations. An anticipated benefit of the approach is to generate thresholds that can be applied directly to forecasts produced by ECMWF’s model (the Integrated Forecasting System; IFS); since the IFS has the same resolution and a similar configuration to ERA5 Land, it should be prone to similar biases. This circumvents a regular issue of missed events caused by applying thresholds based on in-situ data directly to bias numerical weather prediction (NWP) systems. However, given that ERA5 Land employs an older model version that current forecasts, it may be that biases will persist. We evaluate the ability of ECMWF medium-range forecasts to predict past cold wave events, as defined by the tailored EAP thresholds applied to reanalysis data (this is our proxy ‘truth’ data). We use an ‘event based’ approach to contingency table evaluation that focuses on start date, which is important for LRCS: “Knowing the exact date at which an extreme event is starting is quite crucial … to be able to properly plan operations” (Sebongile Hlubi, Anticipatory Action and Readiness Coordinator) and consider whether the forecast could support action at a longer lead time. 9 For issue of Early Warning Messages: “Dissemination to communities will be triggered by LMS issuing a forecast of at least 10°C or less in the Lowlands of maximum day temperature and this forecast temperature remains for 3 or 4 consecutive days. The expected lead-time will be in the range of 4 days” Distribution of clothing and cash transfers will be undertaken: “When LMS issues a forecast of 2°C maximum day temperature or less in the Highlands for at least 2 consecutive days with possibility of moderate snowfall in the Highlands. The expected lead-time will be in the range of 4 days” D3.4 User-driven Evaluation of the I-CISK Climate Services 2.3.1 Coldwave Methodology 2.3.1.1 Historical events database To create the database, we search for impacts attributed to cold waves occurring anywhere across Lesotho, or close to the South Africa-Lesotho border, between 01 Jan 2004 and 31 Dec 2022. LMS and LRCS opinion is split on the inclusion of impact records from South Africa; one deems it acceptable in order to increase the number of event records, whilst the other sees it as not important because the EAP should target Lesotho. In future, if more events occur or the database is further developed, it maybe be beneficial to re-run this analysis with Lesotho-specific events. However, for the present study we retain the South African events. The time-period considered matches the availability of re-forecast data created using the 48r1 configuration of the ECMWF model (the Integrated Forecasting System, IFS; see Section 2.1.2: Re-forecast and reanalysis data for a definition of re-forecasts and discussion on their use). Three main sources contribute to the database: A table of events provided in the EAP, which includes some impact records and spans 2011 to 2022 Grab & Linde (2014), which provides a list of events and their impacts between 1982 and 2012 Google search results for ‘cold’, ‘snow’, ‘Lesotho’ and ‘South Africa’, including media reports and other articles (similar methods are used in e.g. Lines et al., 2017) Start and end dates are recorded as accurately as possible from the reports, with a focus on the date of the impacts rather than the date the report was written. However, there remains a degree of uncertainty in event timing especially, for example, because news articles are often written when events are ongoing. Unfortunately, most records are not detailed enough to pinpoint exact impact locations, so the database considers Lesotho as a whole. For each event identified, impacts are categorized as minor, significant or severe according to Table 2, which we created as part of this work based on i) the Hazards Impact Framework developed by the UK Natural Hazards Partnership (Hemingway & Gunawan, 2018), ii) WMO guidelines on impact based forecasts and warnings (WMO, 2021) and iii) the Flood Forecasting Centre’s impact categories (Flood Forecasting Centre, 2022). Table 2 Impact categories and severities. Developed as part of this work, following methods referenced in the text. Danger to life Livelihood Infrastructure Transport Minor Individual risk for vulnerable/unfamiliar Isolated impacts on livelihood Localized and/or short-term disruption to utilities and services Isolated or prone routes disrupted/unfamiliar users affected Significant Danger to life with localized support required Localized impacts on livelihood Disruption to utilities and services, damage to structures Widespread disruption to travel, some areas temporarily inaccessible Severe Loss of life Rescue operations required Substantial and/or widespread impacts on livelihood Widespread and/or prolonged disruption due to loss of utilities and services, extensive damage to structures Widespread and long duration disruption, large numbers/whole communities cut off 10 D3.4 User-driven Evaluation of the I-CISK Climate Services 21 events are identified, with 8 categorized as having severe impacts, 5 significant, and 8 minor (Table 3). It is acknowledged that some events, particularly at the minor impact level, will be missed because we are using sporadic, incomplete records (especially in the case of e.g. media reports) and are located out-ofcountry. Impactful events were found in each month from April to October, with July and August seeing the highest number of severe and significant events. Work by Grab and Linde (2014) and Grab et al (2017) found that in July and August (classified by them as the ‘Mid-Season’ for snowfall), snow event duration tended to be longer, which may explain the higher impact levels. Table 3 Summarised impacts database; an expanded version can be found in Appendix 1. Start Date End Date Impact Category Impact Type Impacts Reported 28/07/2004 29/07/2004 Severe Danger to life, Infrastructure, Transport Hundreds of motorists trapped in snow Power and telephone services knocked out 06/09/2004 08/09/2004 Minor Danger to life, Transport 15 Tourists trapped in snow 06/09/2005 06/09/2005 Minor Danger to life Tourists trapped in snow 03/08/2006 05/08/2006 Severe Danger to Life, Transport Airborne rescue operation of people trapped for 3 days in snow; reported numbers vary between 8 and 39. Roads closed. 16/08/2006 16/08/2006 Severe Danger to life, Transport Fatalities, rescue operations and closed roads reported in neighbouring Eastern Cape Province South African. Impacts in Lesotho not detailed. 21/05/2007 26/05/2007 Severe Danger to life, Infrastructure 21 deaths reported in South Africa; impacts are not detailed for Lesotho but The Drakensberg Mountains and neighbouring KwaZulu Natal are mentioned. Snow on airstrips reported to have hindered the work of missionary pilots in Lesotho 27/06/2007 28/06/2007 Severe Danger to Life, Transport Rescue operation required from vehicles and trucks on the Lesotho SA border. 19/09/2008 22/09/2008 Minor Infrastructure Electricity went out during snowstorm in Mokhotlong 08/06/2011 10/06/2011 Significant Transport Roads closed for several days; Tourists trapped in Lesotho snow 11 D3.4 User-driven Evaluation of the I-CISK Climate Services 25/07/2011 26/07/2011 Severe Danger to life, Livelihood, Transport Air rescue operation required; Loss of communications; Damage to buildings; Major disruption to roads (and shipping in SA). 14/07/2012 15/07/2012 Severe Danger to life, Transport Rescue from vehicles required; [Two deaths in South Africa, but none reported in Lesotho] 07/08/2012 10/08/2012 Significant Livelihood, Transport Roads closed; Mine unable to operate 23/07/2016 27/07/2016 Severe Danger to life, Livelihood, Transport Eight human lives lost; Livestock deaths; Air rescue operation required 12/05/2017 12/05/2017 Minor Transport Roads impassable in Maseru 31/05/2018 31/05/2018 Minor Transport Roads closed 03/10/2018 04/10/2018 Significant Livelihood, Transport Thousands of livestock dead; At least one road closed 25/04/2020 29/04/2020 Minor Transport Road restrictions 10/06/2020 16/06/2020 Minor Transport Road infrastructure blocked for a week 14/08/2021 15/08/2021 Significant Infrastructure, Livelihood, Transport Communities cut off; Power outages; Tourism industry disrupted; Road closures and traffic disruption 26/08/2021 27/08/2021 Significant Livelihood Livestock deaths 21/05/2022 21/05/2022 Minor Transport Disruption on roads 2.3.2 Reanalysis and re-forecast data ERA5-Land reanalysis data (9 km resolution) are used for tailoring EAP thresholds and as ‘truth’ for reforecast evaluation. ERA5-Land is preferred to ERA5 (31 km resolution) because it has the same resolution as the ECMWF medium-range ensemble and its snow depth is more accurate in regions analogous to Lesotho i.e. with complex orography between ~1500 and ~3000 m amsl (above mean sea level) and a lack of observations contributing to ERA5 data assimilation (S. Grab et al., 2017; Muñoz-Sabater et al., 2021). 12 D3.4 User-driven Evaluation of the I-CISK Climate Services For this analysis, we use an ensemble re-forecast which consists of a single control member and 10 perturbed members. The control member is produced by generating initial conditions which result from the assimilation of observation data into an initial estimate of the atmospheric state, the result is used to generate a single forecast. Perturbations are applied to the initial conditions to reflect errors and uncertainties in the observations and initial estimate, a forecast is then generated for each perturbation. We analyse the control member separately to the control combined with the perturbed members, to understand what benefit can be gained from an ensemble approach. For the control evaluation, forecast event start dates are directly identified using the tailored action trigger threshold. For the ensemble we need to calculate the probability of an event starting. Two different methods are tested which we call ‘stack’ and ‘start’. For probability levels between 10 and 100%, in increments of 10%, we identify ‘stack’ event starts when the percentage of members having a cold wave event in progress first exceeds a given level. For ‘start’ we take only the percentage of members with an event starting on each day. This is illustrated in Figure 9. All re-forecasts issued in April to October, 2004-2022, are analysed, not just those issued in the run-up to ‘true’ events. This is crucial for a fair reflection of false alarm counts, which is very important for operational forecasting. We first assess whether the start dates of cold wave events in ERA5-Land were predicted exactly by the reforecasts. For the control, a hit is scored if start dates match exactly, a miss occurs when a ‘true’ event is not forecast, and a false alarm when an event is forecast but was not observed. Lead time is recorded as the difference between the re-forecast run time and the start of the ‘true’ event for hits and misses, and as the difference between the re-forecast run time and the start of the forecast event for false alarms. While the ideal forecast would be one that can always predict the exact date on which an extreme event will start, in reality this is not always the case, with exact prediction becoming more challenging as lead times are extended. Since there is still value in a forecast that is ‘nearly’ correct, we also count forecast start dates that occur within ±3-days of a ‘true’ start date as ‘too early’ or ‘too late’ and record the number of days by which they are out. Events that are wrong only by one day are then used to make a secondary, adjusted contingency table assessment (i.e. misses become hits if they were within ±1 day of the real event). The same method is applied to the ensemble at each probability level between 10 and 100 in 10% increments (e.g. if the ensemble probability of an event start is 30% or above on a given day, and a ‘true’ event starts on the same day, the ensemble scores a hit at the 30% level, and so on). 19 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 9 Illustration of identifying events in each ensemble member of a re-forecast (red shading) and two different methods of calculating the probability of a start date from the individual ensemble members. The start method takes the percentage of members that actually start an event on each day, whereas the stack method takes the number of members where an event is occurring (so long as it is the first time that probability has been reached). Member zero is the control run and is included in the ensemble for evaluation purposes. 20 D3.4 User-driven Evaluation of the I-CISK Climate Services The method appears simple but there are adjustments that have to be made to ensure that the re-forecasts are not penalised due to the following aspects of the evaluation methodology which cannot ever be achieved: 1) Re-forecasts cannot be verified at 15/14-days lead time because the 2/3-day persistence requirement of the EAP action trigger threshold cannot be applied. 2) If a ‘true’ event starts before the re-forecast was run and then persists into its range, the re-forecast is not penalised if it starts a new event during that time because it could not have known the ‘real’ start date. 3) Time ‘double penalties’ occur when the model predicts an event that starts either too late or too early (scoring both a miss, when the ‘true’ event starts, and a false alarm, when the re-forecast event starts). To avoid this, we define a buffer of ±3 days, within which we consider the start dates of ‘true’ and re-forecast events to be related to the same event (i.e. they make a truth-forecast pair). Re-forecast events having start dates within this buffer but that are too early or too late score only a miss, not a miss and a false alarm. If more than one re-forecast event occurs within the range, we take the one closest to the ‘true’ start date. 40 unique ERA5-Land cold wave events are evaluated, but because typically 3 to 4 forecasts will cover each unique event, this will result in a larger sample of 136 forecast events (Figure 10). Each lead time will not contain the same combination of past events. Different numbers of forecast events also occur at each lead time, for the same reason but with the added complication that events will not be forecast at the same time (or at all) in every run. This creates noise in the contingency table results, especially for frequency bias. A further complication is that forecast issue frequency will affect the number of false alarms and hits/misses in slightly different ways – as there are only a finite number of real events that can be detected it is possible that false alarm counts could increase more than hit/miss counts with a higher issue frequency. This may mean that results generated from re-forecasts, which are essentially ‘issued’ twice per week might, do not directly map to operational forecasts that are issued daily – however, the general performance trends should be the same. Figure 10 Number of ‘true’ and control re-forecast events at each lead time. Note that the actual number of real events doesn’t change, but the number of times they occur in a re-forecast does. 21 D3.4 User-driven Evaluation of the I-CISK Climate Services 2.3.2.3 User feedback Our user-centred evaluation was presented to LRCS and LMS via Microsoft Teams in March 2025. We subsequently collected feedback on four main topics using Microsoft Forms: ‘Methodology’, ‘Communication/Presentation’, ‘Usefulness’ and ‘Future’. The results of the feedback are summarised later in sections 2.3.3.7 (user feedback on ensemble evaluation) and within the discussion in Section 2.3.4. 2.3.3 Coldwave Evaluation Results 2.3.3.1 Tailoring the EAP action trigger thresholds Results are presented below for the action trigger threshold tailoring, where we compare events, detected in ERA5 Land using a range of maximum temperature and snow accumulation thresholds with historical, to impactful events from our database. 2.3.3.2 Lowlands EAP trigger threshold The EAP action trigger for the Lowlands is ‘maximum temperature of 10°C or less sustained for at least 3 consecutive days’ (Text Box 1). We test the ability of temperatures in the range 8-12°C to detect past impactful events. Threat scores are low across the 8-12°C range in the Lowlands, with a maximum of 0.21 scored at 9°C (Table 6). Importantly, the low scores are driven not only by high false alarms (which are expected because our historical events database is unlikely to be complete) but by substantial numbers of misses (e.g. using the 10°C threshold, more than half of past events were not detected). This does not particularly improve if the 3 consecutive days below threshold criterion is relaxed to 2 days (Table 6). Table 6 Threshold tailoring for the Lowlands. Action trigger threshold: Daily Maximum temperature (℃) Action trigger threshold: Event duration (days) Hit Miss False Alarm Threat Score 12 3 13 8 73 0.14 11 3 11 10 52 0.15 10 3 10 11 34 0.18 10 2 12 9 66 0.14 9 3 8 13 18 0.21 8 3 6 15 12 0.18 The 10°C Lowlands EAP threshold triggers the issue of early warning messages, which is a ‘no regret’ action for LRCS; it is not costly and raises awareness. A good threshold for this type of action should not miss any events, even if the cost is a high number of false alarms. This is not the case for the original EAP Lowlands action trigger threshold or any variation that we have tested here. Therefore, we do not consider this threshold further in this study. 2.3.3.3 Highlands The EAP action trigger for the Lowlands is ‘maximum temperature of 2°C or less sustained for at least 2 consecutive days with possibility of moderate snowfall’ (Text Box 1). We test the ability of temperatures in the range 1-3°C combined with snow accumulation in the range 0.5-2 cm to detect past impactful events. Threat scores for the Highlands are relatively low across the board (Table 7), although better than for the 22 D3.4 User-driven Evaluation of the I-CISK Climate Services Lowlands, with a maximum threat score of 0.35 at 1°C/1 cm (13 hits, 8 misses and 15 false alarms). The original 2°C EAP threshold, with 0.5 cm of snow accumulation, produced a similar threat score of 0.31, with more hits (15), fewer misses (6) and more false alarms (28). Including 0.5 cm of snow accumulation reduces the number of false alarms from 36 to 28 compared to using temperature alone (Table 7). Table 7 Threshold tailoring for the Highlands Action trigger threshold: Daily Maximum temperature (℃) Action trigger threshold: Daily Maximum snow accumulation (cm) Hit Miss False Alarm Threat Score 3 0.5 16 5 34 0.29 3 1 15 6 32 0.28 3 2 13 8 21 0.31 2 NIL 15 6 36 0.26 2 0.5 15 6 28 0.31 2 1 14 7 26 0.3 2 2 12 9 18 0.31 1 0.5 13 8 18 0.33 1 1 13 8 16 0.35 1 2 11 10 15 0.31 Despite the low threat scores, these results are encouraging because events will be missing from our historical impacts database, meaning we expect some false alarms. Unlike the Lowlands trigger thresholds, those we have tested in the Highlands show a better capacity to detect events, for example with 15/21 events detected by the 2°C, 0.5 cm combination. This original Highlands action trigger threshold seems suitable for its intended purpose of identifying impactful events for which anticipatory action is required. 2.3.3.4 Forecast evaluation results Event-based contingency table analysis of ECMWF medium range re-forecasts from April to October, 20042022, relative to ERA5-Land ‘truth’ data, is presented below. The tailored EAP action trigger threshold ‘maximum temperature falling below 2°C with 0.5 cm of snow accumulation, sustained for at least 2 days’ is used to identify cold wave events. Whilst this threshold did not have the absolute highest threat score of the combinations we trialled (see previous section) it did have the lowest miss count of combinations with threat scores >0.3. 2.3.3.5 Control (deterministic) evaluation For all months and lead times (except 14 days), the re-forecasts predict cold wave events too frequently relative to ERA5 Land (frequency bias >1, dashed black line in Figure 11). This tendency is stronger in September (bias=3.86) and October (bias=3) than in other months (bias=1.56-2.07). Frequency bias is noisy when plotted against daily lead time, most likely due to the 3to 4-day frequency of re-forecast issue. The 23 D3.4 User-driven Evaluation of the I-CISK Climate Services noise is reduced if lead times are grouped (Figure 11 centre and left) and overall values range from 1.17 at 14 days lead time to 2.25 at 5-8 days, again showing the tendency to over-forecast. Figure 11 Frequency bias by month (left), daily lead time (centre), and grouped lead times (right) for the control member of ECMWF medium range re-forecasts. When re-forecast event start dates are required to match exactly the observed start date, the threat scores are low (Figure 12), ranging from 0.02 at 12 days lead time (1 hit, 9 misses, 30 false alarms) to a maximum of 0.48 at 1 day (11 hits, 1 miss, 11 false alarms). There is a general trend toward better threat scores at shorter lead times; threat score = 0.34 at 1-4-days (29 hits, 19 misses, 37 false alarms), 0.12 at 5-8 days (13 hits, 31 misses, 68 false alarms), and 0.06 at greater than 8 days (11 hits, 55 misses, 106 false alarms; acknowledging more lead times are included in this category). Therefore, the forecasts at longer lead times cannot predict the exact event start date, whilst at shorter lead times the forecast performance is better, but users should be aware only half of the forecast event start dates were correct. Do start dates match exactly? If not by how much? What if we allow +- one day? Figure 12 Hit, miss and false alarm counts and threat scores for the case that start dates need to match exactly (left) or can be within +-1 day (right). Both are shown for daily and grouped lead times. The plot in the centre shows counts of how many days out ‘near misses’ were (if the forecast was within 3 days). To better understand the challenges with predicting the correct start date of an event, we use the 3-day buffer methodology described above. Within the ±3-day buffer we set for matching ‘observed’ and forecast 24 D3.4 User-driven Evaluation of the I-CISK Climate Services events, we find the majority of ‘near misses’ are only 1 day out and the re-forecasts are generally predicting the events to start too early (Figure 12). Allowing the model to score a hit for forecasting event start dates that are within ±1 day of the observed event increases the range of threat scores to 0.03-0.5 (threat score = 0.03 at 13 days lead time with 1 hit, 11 misses, 21 false alarms and threat score = 0.5 at 2 days lead time with 12 hits, 0 misses, 12 false alarms, respectively). The biggest benefit is felt between 2and 4-days lead time, e.g. for the 4-days lead time the score increases from 0.28 (when applying the exact start date agreement) to 0.44 (when the ±1 day flexibility is applied). The threat scores were again better at shorter lead times, meaning that the forecasts have better performance at shorter lead times. However, the prevalence of false alarms, even at shorter lead times, caused by over forecasting limits the overall value of the forecasts (Figure 12). The over-forecasting of cold wave events is caused by both maximum temperature and snow accumulation threshold exceedances being predicted too often on the highest ground of the Lesotho Highlands (Figure 13 and Figure 14; see Figure 5 for model topography). Whilst the opposite is true on ground with lower elevation, our use of the max/min spatial sampling method means detection of cold waves is largely dictated by the coldest/snowiest grid squares. Figure 13 Frequency bias for 2°C maximum temperature threshold exceedances. This was calculated for all our reforecasts by comparing the number of instances of temperature at or below threshold to the number of instances on equivalent dates in ERA5 Land. 25 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 14 Frequency bias for 0.5 cm snow accumulation. This was calculated for all our re-forecasts by comparing the number of instances of snow accumulation at or above threshold to the number of instances on equivalent dates in ERA5 Land. Exacerbation of the model’s tendency to over-forecast is caused by our use of the max/min spatial sampling method and suggests that considering a greater portion of the Highlands area might improve performance. We re-ran the contingency table analysis using the quantile method (taking the 0.25/0.75th quantiles of maximum temperature and snow accumulation for comparison to the tailored EAP threshold), which did reduce the frequency bias but did not improve the threat score, which decreased at most lead times (Figure 15). We therefore continue using the max/min method for the ensemble evaluation. Different threshold values might be needed in combination with this spatial sampling method for any improvement in skill to be seen. Such an investigation is beyond the scope of this study, but could be an area for future work. Figure 15 Frequency bias and threat score when the ‘quantile’ spatial sampling method is used to define cold wave events. It is worth noting in Figure 14 that by calculating snow accumulation at 1 day lead time by using ERA5 Land to provide a ‘previous day’ snow depth value we have likely further exacerbated the number of false alarms. 26 D3.4 User-driven Evaluation of the I-CISK Climate Services This is reflected in the stronger frequency bias gradient at 1-day lead time vs. 2 days lead time (Figure 14). However, the general tendency to over-forecast even at this shortest lead time is probably genuine since we see it reflected in the temperature bias at day 1 (Figure 13) and it still occurs in snow accumulation at 2 days lead time where the snow accumulation calculation uses only re-forecast snow depth (Figure 14). 2.3.3.6 Ensemble evaluation Before analysing the ensemble re-forecast performance by considering threat scores, an important question that we need to answer is ‘what probability level should we be looking at?’ This is also key for operational use of ECMWF ensemble forecasts, assuming the results of the 11-member re-forecast ensemble translate to the 51-member operational ensemble. Frequency bias (Figure 16, top) is a useful indicator – where it approaches 1 the forecast is predicting events with the correct frequency. Table 8 summarises the probability levels where frequency bias approaches 1 at each lead time, with corresponding threat scores for the case where hits are scored for events correct to within ±1 day lead time. We focus on the ±1 day evaluation because, as for the control, this significantly improves ensemble performance (Figure 16). For both methods of defining start event probability, there is a strong variability of frequency bias with lead time and probability, with values near 1 occurring at lower probabilities when lead times is longer and at higher probabilities when lead time is shorter. This is expected from a model that becomes increasingly confident as lead time shortens. Of the two methods for defining event start date probability from the ensemble, the ‘start’ method sees frequency bias approaching 1 at lower probability per lead time (p=40 to 80% for 2-4 days lead time ) than the ‘stack’ method (p=70 to 90% for 2-4 days lead time), although this yields little difference in threat score (Table 8). At lead times of 7 days plus, threat scores at probabilities where frequency bias approaches 1 are low and indicate little skill using either method. Overall, there is no strong benefit to using one method over the other, but the most useful probability to look at when trying to forecast events will change depending on which is employed. 27 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 16 Frequency bias (top row) and threat score when start dates are required to match exactly (middle row) and when a ±1 day tolerance is allowed (bottom row) for ensemble re-forecasts. Results for the ‘stack’ method of defining event start probabilities is shown in the right column and for the ‘start’ method in the left column. 28 D3.4 User-driven Evaluation of the I-CISK Climate Services collecting and categorizing impacts was seen as something that could be useful more broadly, for example for different hazards, and both LRCS and LMS said that they might continue to use the method in future. The newly developed impacts table was seen by both LMS and LRCS as applicable to cold wave events in Lesotho, but users would also like to see other sectors such as health and education included. Using historical impactful events to evaluate the EAP thresholds was very much liked, and seen to help gather information on whether it is necessary to prioritize the weather hazard by looking at its actual impact on the ground. Both threshold evaluation vs. historical impacts and ECMWF forecast evaluation were seen as potentially useful for the cold wave EAP, for application to other EAPs/hazards and for forecasting and issuing warnings. Overall, both were seen as ways to clearly agree on the early actions to be taken, and to improve the forecast especially in terms of issuing warnings for extreme weather events. LMS and LRCS liked the contingency table approach to evaluation, and said it was easy to understand, although LMS would also like to have seen other performance measures – in particular Mean Absolute Error and Ranked Probability Skill Score. The contrasting requirements of LMS and LRCS highlights how the type of evaluation technique or metric can vary by user, based on, for example, the type of decision-making being supported or the level of technical interest/training. The use of ERA5 Land as a proxy for historical conditions was seen as partially valid, but the importance of using in-situ observations and of understanding the relationship between them and the reanalysis data was highlighted by LMS. We very much hope that this will be an area of future investigation and collaboration, because it will also provide valuable insights into IFS and ERA5 Land characteristics and biases. Both users would like the code provided as Juypter notebooks ideally with supporting data files, plus the results data and a written report. These resources would help with the EAP development and with forecasting cold wave events. We will endeavour to provide these. 2.4 Lesotho LL – Drought evaluation The Lesotho Red Cross Society (LRCS), in collaboration with the International Federation of Red Cross and Red Crescent Societies (IFRC) and national stakeholders, developed and validated Early Action Protocol (EAP) for droughts in 2022. The EAP relies on seasonal precipitation forecasts to trigger anticipatory actions such as early warning dissemination and unconditional cash transfers. These measures, coordinated with government and disaster management agencies, aim to mitigate drought impacts on vulnerable communities. The Lesotho Meteorological Services (LMS) is mandated to produce national forecasts for humanitarian operations. LMS’s seasonal precipitation forecast, derived from downscaled global models and local observations, is recognized in the country for its accuracy and reliability. The forecast for the rainy season (October-April) is typically released in September and serves as the official seasonal outlook that guides actions by national stakeholders. However, global forecast products can provide value by offering earlier predictions (as early as July). This enables LRCS to initiate informal preparedness, enhancing response time and the EAP's overall effectiveness. The analysis presented below aims at assessing the accuracy of global forecasts compared to LMS’s national forecast to determine the most suitable products for LRCS operations. The rainy season in Lesotho runs from October to April, with the EAP trigger based on cumulative precipitation in December, January, and February (DJF). This analysis focuses on these three months. 35 D3.4 User-driven Evaluation of the I-CISK Climate Services 2.4.1 Lesotho Drought Methodology 2.4.1.1 Creating a database of historical drought impacts To validate forecasts, three observed drought datasets were initially considered (Figure 21): 1. EAP observational records (2011-2020): A dataset created during the development of the Early Action Protocol (EAP), consisting of binary records that indicate whether a drought occurred in the DJF period. The data was provided by the Lesotho Meteorological Services and analysed by various stakeholders involved in the EAP process. 2. Droughts identified in literature (2001-2022): A set of drought years identified through a literature review, particularly from Kamara et al. (reference: https://doi.org/10.1371/journal.pone.0212994) which reports prolonged periods of below-normal rainfall leading to crop damage, reduced stream flow, and low water reservoirs. While the literature specifies droughts within this period, it does not always align with the DJF months. However, given that Lesotho’s rainy season typically spans from October to April, it is assumed that the droughts mentioned likely occurred during or just outside these months. For the purpose of this analysis, these drought years are treated as occurring within the DJF period and were incorporated into the analysis as observed drought years, which were then compared against forecast datasets. 3. HydroGFD3 dataset (1995-2023): A dataset of bias-adjusted reanalysis data for daily precipitation and temperature, designed for large-scale hydrological modelling. It provides global coverage at a 25 km resolution and is updated in near-real time. Figure 21 Observed drought years across different datasets for the rainy season in Lesotho. The heatmap displays drought occurrence by year (x-axis) and observational data source (y-axis), with red cells indicating years classified as drought and light grey cells. Among these datasets, we ultimately chose to focus on the second one, derived from literature. Ideally, we would have used the EAP observations, but the data available is too limited. In contrast, the literature-based dataset is larger which allows for a more robust analysis, and it coincides with the three observed EAP events. 2.4.1.2 Assess decision threshold A 25% probability of DJF (December-January-February) precipitation falling in the lower tercile of the climatology, defined from reforecasts, across Lesotho was identified as the threshold above which drought conditions are likely. This threshold was established through the EAP development process, which involved extensive consultation with multiple stakeholders. It reflects a deliberate effort to incorporate local knowledge and professional expertise, drawing on the lived experiences of communities and the insights of national and regional experts who understand the local context and climate risks. This inclusive approach enhances the accuracy and contextual relevance of the threshold, while also fostering stakeholder ownership and broad support for early action implementation. 36 D3.4 User-driven Evaluation of the I-CISK Climate Services We chose to retain the same threshold in our analysis because it was the one defined in the EAP. This ensures consistency and allows for a meaningful comparative analysis when using alternative forecast sources. 2.4.1.3 Forecast Evaluation Four forecast datasets were evaluated: 1) precipitation forecast records (2011-2020): This dataset comprises seasonal climate forecasts generated as part of the EAP development process. LMS uses the Climate Predictability Tool to generate high-resolution seasonal forecasts through a multi-model ensemble approach (ENACTS), which blends global model outputs with local CHIRPS data. As direct access to their data was not possible, we relied on the forecasts they produced for the analysis presented in the EAP. The forecast is produced and disseminated in September. 2) ECMWF SEAS5 precipitation forecast (1995-2022): The European Centre for Medium-Range Weather Forecasts (ECMWF) developed the SEAS5 system, which represents the fifth generation of its seasonal forecasting system. Operational since November 2017, SEAS5 provides global seasonal forecasts with a focus on improving predictions in the tropics. The system offers probabilistic forecasts for various climate variables, including precipitation, temperature, and atmospheric pressure, with lead times ranging from one to seven months. The dataset was used from 1995 to 2022 and was downloaded through the Copernicus Climate Change Service's multi-system seasonal forecast. 3) SMHI downscaled SEAS5 precipitation forecast (1995-2022): The Swedish Meteorological and Hydrological Institute (SMHI) has downscaled the ECMWF SEAS5 forecasts to provide higher-resolution seasonal forecasts tailored for specific regions. This downscaling process involves statistical methods to refine the coarse-resolution (~36 km) SEAS5 outputs, enhancing their applicability for regional impact assessments and decision-making. The downscaled dataset covers the period from 1995 to 2022 and includes detailed forecasts of climate variables at a finer spatial resolution. 4) IRI seasonal precipitation forecast (2017-2023): The International Research Institute for Climate and Society (IRI) at Columbia University maintains a comprehensive data library that includes seasonal climate forecasts. These forecasts are derived from the U.S. National Oceanic and Atmospheric Administration (NOAA)’s North American Multi-Model Ensemble Project (NMME), which combines outputs from multiple climate models to produce probabilistic predictions of climate variables. The IRI Data Library provides access to these forecasts, expressed using tercile categories (below normal, near normal, above normal), as well as user-selectable thresholds. The dataset spans from 2017 to 2023. 2.4.1.4 Skill assessment The skill analysis evaluates the performance of various forecast models by comparing each forecast against the observed drought dataset. The purpose of the analysis is to determine the accuracy and reliability of the forecasts in predicting drought events. To assess the forecast performance, a contingency table is used, which categorizes the outcomes into four distinct categories: hits, misses, false alarms and correct negatives (see section 1.1.2 for definitions). Several performance indicators are then calculated to evaluate the forecast's ability to predict drought events: Probability of detection, false alarm ratio, accuracy, bias and threat score (Table 1). 37 D3.4 User-driven Evaluation of the I-CISK Climate Services Each of these skill metrics offer insights into different aspects of forecast performance. A high POD with a low FAR is generally indicative of a strong forecasting model, as it reflects both effective detection of droughts and minimal false alarms. However, accuracy should be interpreted cautiously, especially if the dataset contains a high number of true negatives, which can artificially inflate the metric. Bias is also crucial for identifying potential tendencies of the model to overor under-predict drought events, providing valuable insights into the model’s reliability over time. 2.4.2 Results The results of the assessment are shown in Table 9 and Figure 22. Table 9 Forecast skill metrics for all products considered, for the DJF period across Lesotho. Observational data described in Section 2.3.1 is used as the reference ("truth"). Note that not all forecast products are available at all lead times: the EAP forecast is only available from September, while the IRI forecast begins in August. Month of issue Product name Hit s Misse s False alarm s POD FA R Accurac y Bia s TS # years July SEAS5 Forecast 11 3 6 0,79 0,35 0,67 1,2 0,55 27 July Downscaled SEAS5 Forecast 3 11 1 0,21 0,25 0,56 0,29 0,20 27 August SEAS5 Forecast 9 5 6 0,64 0,4 0,59 1,1 0,45 27 August Downscaled SEAS5 Forecast 3 11 1 0,21 0,25 0,56 0,29 0,20 27 August IRI Forecast 2 0 2 1 0,5 0,6 2 0,50 5 September EAP Forecast 3 4 0 0,43 0 0,6 0,43 0,43 10 September SEAS5 Forecast 9 5 8 0,64 0,47 0,52 1,2 0,41 27 September Downscaled SEAS5 Forecast 3 11 1 0,21 0,25 0,56 0,29 0,20 27 September IRI Forecast 2 0 2 1 0,5 0,6 2 0,50 5 Figure 22 Threat Score by forecast product and lead time for the DJF period across Lesotho. Bars represent the skill of each product in detecting events while accounting for missed events and false alarms. Values are derived from the 38 D3.4 User-driven Evaluation of the I-CISK Climate Services results presented in Table 9 Forecast skill metrics for all products considered, for the DJF period across Lesotho. Observational data described in Section 2.3.1 is used as the reference ("truth"). Note that not all forecast products are available at all lead times: the EAP forecast begins in September, and the IRI forecast in August. 2.4.3 Discussion: Drought evaluation in the Lesotho Living Lab This analysis evaluated the performance of multiple seasonal precipitation forecast products in predicting drought events in Lesotho, with the goal of identifying the most suitable dataset to support the Lesotho Red Cross Society's preparedness action with a lead time greater than the one that the official seasonal forecast can offer. Forecasts were assessed using key skill metrics (Probability of Detection, False Alarm Ratio, Accuracy, Bias, and Threat Score) against a literature-derived dataset of observed droughts. The focus was on the DJF (December-January-February) rainfall period, which serves as the critical window for triggering early actions. Table 9 presents a detailed breakdown of these forecast performance metrics across different issue months and forecast products, while Figure 22 visualizes the Threat Score, offering a direct comparison of each product’s ability to correctly detect droughts while minimizing missed events and false alarms. Overall, the analysis reveals that none of the forecast products exhibit strong or consistently high skill across all metrics and lead times. However, the ECMWF SEAS5 forecast (not downscaled) performs on par with or better than the other products, especially when considering its broader historical range, earlier availability, and open-access format. SEAS5 demonstrates relatively high Threat Scores across lead times (see Figure 22), particularly when issued in July and August. Although the IRI forecast shows high skill for the limited period available, it is only issued starting in August and covers a significantly shorter time span (2017–2023), which constrains its reliability for statistical analysis. By contrast, SEAS5 is available from July, providing a longer lead time that is operationally advantageous for early preparedness activities. It is also publicly available through the Copernicus Climate Data Store, making it suitable for integration into anticipatory action platforms such as the LRCS’s Impact-Based Forecasting Portal. The downscaled SEAS5 product, while offering finer spatial resolution, does not demonstrate improved performance over the original SEAS5 and consistently underperforms across all lead times. Similarly, the EAP national forecast shows acceptable skill but is only issued in September, limiting its utility for early preparedness. Given these findings, SEAS5 stands out as being the most appropriate forecast product for earlier preparedness in Lesotho. Its reasonable forecast skill, early release, open access, and consistent availability make it well-suited for informing proactive drought preparedness decisions before the rainy season. However, it should be used in tandem with the forecast from the national agency once it becomes available, this would give added confidence given the noted trust in the national forecasts. Several limitations of the analysis should be acknowledged: 1. Lack of severity and spatial detail in drought observations: The observed dataset used is binary (drought/no drought) and does not capture the intensity or precise location of drought events. 2. Country-level aggregation of forecast data: All forecast data were averaged across the entire country, which may obscure regional variations in precipitation and drought impact. This could have reduced the benefit of using downscaled forecasts, future work could investigate how to use these data at a sub-national scale. 3. Potential influence of climate change trends: Long-term shifts in climate patterns may affect the historical validity of forecast skill assessments. 39 D3.4 User-driven Evaluation of the I-CISK Climate Services 4. Short data coverage for some products: Products like the IRI forecast are only available for a limited number of years, reducing the robustness of skill evaluations. These constraints highlight opportunities for improving future assessments, such as incorporating locally validated impact data, refining thresholds, and examining regional forecast performance. Nonetheless, SEAS5 currently offers the most practical and operationally valuable option for supporting drought early action in Lesotho. 40 D3.4 User-driven Evaluation of the I-CISK Climate Services 3 The Rijnland Living Lab (Netherlands) 3.1 Summary Seasonal forecasts from E-HYPE of streamflows were evaluated to predict the occurrence of low flows in the Rhine river basin at Lobith, the Netherlands between 1994-2015. Results found an overprediction of low flows for the Rhine at Lobith when compared to observed data. Despite this, the seasonal forecasts could support drought alerts up to lead times of 1-2 months. User feedback identified that the contingency table based skill scores helped them better understand forecast performance than conventional statistical approaches, such as the ROC (Relative Operating Characteristics) curve. 3.2 Introduction The Rijnland Living Lab is located in the Netherlands, within a marine west coast climate zone (Köppen classification: Cfb), which is characterised by mild temperatures and relatively uniform rainfall throughout the year. However, between April and July, the region experiences a precipitation deficit due to the combination of higher temperatures and reduced rainfall. This seasonal imbalance increases the chance of dry conditions. Climate projections suggest that the region will face more frequent extreme rainfall events and prolonged dry periods in the future, further exacerbated by temperature and sea level rise. These changes pose challenges to freshwater availability and increase the risk of saline water intrusion — key concerns for regional water managers (Masih et al., 2022). Rijnland is a small but highly managed sub-catchment at the downstream end of the Rhine delta, covering approximately 1,000 km², with about 72% of the area lying below sea level due to historical peat excavation, land reclamation, and continued soil subsidence. Water is actively pumped out toward the Rhine, canals, or the North Sea using an advanced drainage system. The region supports a mix of land uses, including agriculture and urban areas with historic cities. The Rijnland Water Board is responsible for managing surface water quantity and quality, operating pumps, weirs, and inlets to balance irrigation, flood protection, and drinking water needs. The organisation coordinates closely with municipalities, neighbouring water boards, and sectors like agriculture, industry, nature conservation, and navigation. The Lobith discharge station on the Rhine River, where water enters the Netherlands, is a key reference point for national drought monitoring. Water levels at Lobith influence navigation, water distribution, and drought preparedness throughout the country, including the Rijnland region (see Figure 23). Downstream of Lobith, water is highly managed through a system of canals and pumps, which makes Lobith a critical control point for water management across the country. 41 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 23 Location of the Rijnland Living Lab in the Netherlands. Considering the region’s vulnerability to dry conditions, the water authorities rely on climate services—such as forecasts—to support water allocation and issue drought alerts. In the Rijnland Living Lab, one such climate service is being evaluated through a co-evaluation process that combines numerical performance metrics with user feedback, following the approach and analysis of Correa (2024). 3.3 Methodology 3.3.1 Climate service and decision-making In this Living Lab, the climate service under evaluation is a (sub)seasonal streamflow forecasting system designed to support drought monitoring and water-related decision-making. The service provides daily discharge forecasts at Lobith with lead times ranging from 7 days to 7 months. These forecasts are particularly relevant from April to September, when freshwater demands increase, and river inflow from the Rhine River is a key factor in regional planning. They are generated using the E-HYPE system developed by SMHI, driven by SEAS5 seasonal meteorological inputs. Observed discharge data from Rijkswaterstaat at Lobith were used as a benchmark. The reforecast dataset spans 1994 to 2015, and the model has a spatial resolution of 215 km², which allows for representation at a national planning scale (see example in Figure 24). Figure 24 Rijnland Climate Service - Streamflow forecasts from i-cisk.dev.52north platform. 42 D3.4 User-driven Evaluation of the I-CISK Climate Services Operational decisions are based on monthly drought alert thresholds pre-defined by water authorities. These thresholds vary by month—for example, 1000 m³/s in April, 1400 m³/s in May, and 1300 m³/s in July— and are used to trigger national drought coordination phases (impact-based). In the Rijnland region, this information is used by water managers, farmers, and recreation stakeholders to anticipate low-flow conditions and adjust water allocation accordingly (Figure 25). Figure 25 Decision-making processes and drought thresholds in the Rijnland LL. Figure 26 shows the monthly drought alert thresholds from April to September period marked by red dots. Note that the threshold is not the same for each calendar month. While average discharge levels generally remain above these thresholds, the 95% confidence interval indicates that extreme low-flow events can happen at any time of the year. Feedback from the Rijnland Water Board reinforced the importance of streamflow forecasts during this period, not only because of hydrological risks but also due to the higher societal and economic impacts of low flows. Figure 26 Average minimum discharge from observed values compared with monthly thresholds. The co-evaluation process focuses on whether the forecasts aligned with actual decision timelines and if they could provide added value for early warning and planning. To evaluate forecast quality and usefulness, 43 D3.4 User-driven Evaluation of the I-CISK Climate Services a co-evaluation process was carried out with two representatives from the Rijnland Water Board. The process combined tailored visualisations with a structured questionnaire covering different dimensions: selfassessment, utility, understandability, coherence, credibility, and engagement (Figure 27). These criteria were adapted from existing literature on climate services and co-produced knowledge and were selected to ensure both technical and practical aspects of forecast use were considered (Correa, 2024). The aim was not only to assess the statistical skill of the forecasts but also to understand how meaningful and usable the information is for real-world decision-making. Figure 27 Co-evaluation framework and criteria (Correa, 2024). 3.3.2 Streamflow Evaluation The quality of streamflow forecasts was assessed using a combination of quantitative metrics and visual inspection. The evaluation was performed with the Ensemble Verification System (EVS) (Brown et al., 2010), selected for its flexible interface and ability to handle large forecast datasets across different thresholds and lead times. The verification focused on the period April to September, when drought management is operationally relevant. During the verification period (1994–2015), 132 forecast–observation pairs were used in the analysis (Figure 28). This number comes from the fact that forecasts are issued once a month and only on days between April and September are kept. Over 22 years, this gives 132 days where forecasts and observations can be compared, and this same set of days is used for every lead time. When we apply thresholds (for example, keeping only days when discharge is above 1400 m³/s), the number of available pairs reduces. That happens because only a few of those 132 days have such high values. As a result, continuous scores like Mean Absolute Error (MAE) or Continuous Ranked Probability Score (CRPS) become less reliable, since they are based only on the remaining pairs that pass the threshold. In the case of Brier Score and contingency tablesbased thresholds for example, the selection of a threshold does not reduce sample size, because it considers a specific probability of the threshold being exceeded, without resampling the data. 44 D3.4 User-driven Evaluation of the I-CISK Climate Services 2. SMHI bias-corrected SEAS5 (bias correction conducted using ‘standard’ HydroGDF observations and a quantile mapping method; Berg et al., 2021) – past forecast data available for evaluation and assessed here 3. SMHI ‘standard’ bias corrected SEAS5 data with geostatistical downscaling by CREAF (Trojer et al., 2024) – past forecast data not available at the time of analysis 4. SMHI ‘bespoke’ bias corrected SEAS5 (bias correction using additional long time series of observed data from AEMET) with geostatistical downscaling by CREAF – past forecast data not available at the time of analysis Both bias correction and geostatistical downscaling adjust ‘raw’ forecasts based on statistical relationships with past observed data and explanatory topographic variables. Bias correction typically adjusts the forecast parameter values, whilst downscaling increases the spatial resolution. The aim is to improve forecast accuracy, but the adjustment requires some degree of processing and support that incurs costs and raises questions around post-ICISK sustainability. SEAS5, conversely, is an open access dataset. Here, we investigate whether the SMHI ‘standard’ bias corrected SEAS5 forecasts provide a tangible benefit in terms of performance over SEAS5 when forecasting values of interest to Olive, Dehesa1 and Dairy farmers in the Los Pedroches region. We define these values of interest using the results of a ‘timelines’ exercise that was run by the Spain-LL leads. Farmers and cooperative extension agents were asked to consider how the weather affects their decision-making throughout the year, and the quantitative values they provided for temperature and precipitation are summarised in Figure 35. The farmers describe temperature and precipitation forecasts as the potential foundation for many decisions, and so we originally set out to evaluate the performance of both. However, we found a mismatch between what the farmers need to know and the 24-hour average temperature provided in the SMHI biascorrected forecasts. Figure 35 (bottom) shows that farmers’ generally need to know about maximum/minimum daily or perhaps day/night average temperatures, which are not available and cannot be calculated from the 24-hour averages. For this reason, we do not feel it is useful to evaluate the biascorrected temperature forecast any further. SEAS5 does produce maximum and minimum daily temperature forecasts, which might provide a user-relevant starting point for future bias-correction. 51 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 35 User feedback on conditions of precipitation and temperature that are ideal for each type of agriculture (green). Upper (temperature) and lower (precipitation) limits are coloured orange and included where provided. Lower temperature limits are in blue. 52 D3.4 User-driven Evaluation of the I-CISK Climate Services For precipitation, the farmers provided values accumulated over time periods of half a month upward (Figure 35 top), which can be easily calculated from the avaialble SEAS5 and SMHI bias-corrected precipitation forecasts. Many value/accumulation periods were highlighted and it is beyond the scope of this work to analyse them all. We focus on four combinations that cover three seasons and two accumulation periods. Whilst generally the farmers provided specific ‘ideal’ values, there is likely to be an ‘adequate’ range around this, which is what we require to evaluate performance in a meaningful way. As a first pass, we set the ‘adequate’ range to be the ideal value plus/minus half (or half the middle of the narrow ideal range provided in some cases). Ideally, we would have sought user input on these ranges, but sadly time has not permitted. The ideal values and corresponding ‘adequate’ ranges are summarised below: 1) 100 mm accumulated precipitation in October [adequate rage set at 50-150 mm]. This is ideal for germination of winter fodder crops planted by the Dairy Farmers. Whilst they would ideally like it all to fall in the first 15 days of October, we begin by analysing the whole month because forecasts such as SEAS5 are generally considered useful only for detecting broader trends. If the forecast is found to be skilful for monthly October precipitation, a future investigation of 15-day totals would be justified. The Dairy Farmers need to know in August/September how much rain will fall in October (1-2 months lead time). 2) 150-200 mm accumulated precipitation in Autumn (September, October and November) [adequate range set at 100-300 mm]. This is the ideal range for the Dehesa farmers who need to know in late August/early September how much it will rain in Autumn (1 month lead time) so they can decide what, if any, type of crop to sow. The results of this analysis may also be relevant to some degree for the Olive farmers, who need 200 mm of rainfall in September and October both for fertilisation and olive growth. 3) 200-300 mm accumulated precipitation in Winter (December, January and February) [adequate range set at 125-375 mm]. This is the ideal range for the Dehesa and Dairy farmers when they are considering water availability for livestock. However, for production of crops, the Dairy farmers said they needed somewhat less rainfall in winter, with 30-40 mm per month being ideal (90-120 mm for winter as a whole). 4) 400 mm accumulated precipitation in Spring (March, April and May) [adequate range set at 200600 mm2]. The Dehesa farmers stated that this is an ideal amount of rainfall for winter cereal crop growth. They need to know in late September how much and when it will rain in Spring (7/8 months lead time), to decide which, if any, crops to grow. These values, and our estimated ‘adequate’ ranges, provide the starting point for user-centred evaluation of SEAS5 and SMHI bias-corrected forecasts in the Los Pedroches region (plus a 50 km buffer). We test the forecast ability to predict observed rainfall in the ‘adequate’, ‘less than adequate’ and ‘more than adequate’ categories in October, Autumn, Winter and Spring between 1993 and 2022. 4.3 Methodology 4.3.1 Data 4.3.1.1 Observations In-situ precipitation observations for 1993-2022 were provided by CREAF after filtering and other postprocessing of AEMET raw data for the Guadalquivir river basin and the part of the Pedroches region that is in 2 During our assessment of historical observations, we found that the spring value provided by the Dehesa farmers in the timelines exercise was outside the range of our observed historical data when averaged over the Los Pedroches region. Therefore, we adjusted the ‘adequate’ range to 100-300 mm (see Results -Historical precipitation). 53 D3.4 User-driven Evaluation of the I-CISK Climate Services the Guadiana river basin region (Figure 34). Here we only use observations from within Los Pedroches, as this is where the LL is focused, plus a 50 km buffer to increase the number of available sites (Figure 36). Figure 36 Annual average rainfall 1993-2022 for sites that are within a 50 km buffer of the Los Pedroches region. Sites 89 and 465 are highlighted as they are considered in the text and their rainfall records are shown in Figure 45 and Figure 46. Precipitation observations are interpolated onto the 0.25 degrees lat/lon grid used in the SMHI bias corrected forecasts, using an inverse distance weighted method and the Digital Earth Australia (DEA) python tools package dea_tools.spatial (xr_interpolate), with maximum distance set to 30 km, k=10, and the default p=1. After interpolation, we retain only grid cells within the Los Pedroches region plus a 50 km buffer, and then calculate the area average for each accumulation period of interest (October, Autumn, Winter and Spring) in each year. These values are the ‘truth’ data for the contingency table assessment described in Methods – Forecast Evaluation (Section 4.3.1.3). In practice, we find that interpolation makes little difference to the eventual precipitation times series compared to calculating the same area averages using only sites with a full set of data in the accumulation period of interest (Figure 37). This is demonstrated by the mean absolute error (Table 11), which ranges between 2 mm for October and 7 mm for Winter and is well within the standard deviation of the accumulated, interpolated observations (43 mm for October and 136 mm for winter). 54 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 37 Comparison of observed and forecast timeseries for Los Pedroches average precipitation Seasonally (top), Annually (bottom left) and in October (bottom right). Table 11 Mean absolute error and maximum/minimum differences between observed raw and observed interpolated precipitation accumulation averages across Los Pedroches. Mean, maximum, minimum and standard deviation of both interpolated and raw observed datasets are also provided for comparison. Accumulated Precipitation (mm) October Autumn Winter Spring Interpolated mean 70 172 183 156 Interpolated standard deviation 43 72 136 68 Interpolated maximum (year) 178 (2003) 338 (2012) 584 (2009) 338 (2018) Interpolated minimum (year) 6 (1995) 68 (1998) 32 (2011) 39 (1995) Interpolated vs. Raw Mean Absolute Error 2 5 7 3 Interpolated vs. Raw Maximum Difference 11 20 32 9 Interpolated vs. Raw Minimum Difference -5 -10 -8 -11 Interpolated vs. Raw Abs. Standard Deviation 2 4 7 3 55 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.3.1.2 Forecast data The ECMWF seasonal forecast ‘SEAS5’, in its current configuration, is a 51-member ensemble (26 members prior to 2017) with a resolution of 36 km. Forecasts are produced on the first of each month and run out to 7 months ahead (lead time). The SMHI bias corrected forecasts are derived from SEAS5 and so have the same number of ensemble members, original resolution, and run time/lead time. However, they were provided on a 0.25 by 0.25 lat/lon grid. We downloaded SEAS5 ‘total precipitation’ reforecasts from 01 January 1993 to 31 December 2022 and regridded them to match the 0.25 degree lat/lon grid of the SMHI bias corrected forecasts. 6 hourly timestep data for the control and all ensemble members is converted to 24-hour accumulations. SMHI provided bias corrected 24-hour precipitation forecasts, produced using the MIdAS bias adjustment method and ‘hydroGFD3’ reference dataset (Berg et al., 2021 and Berg et al., 2022). The SEAS5 data evaluated here are the underlying dataset for these bias corrected forecasts. For the control and each ensemble member of both the SEAS5 and SMHI bias corrected forecasts, we calculate accumulated precipitation in each grid square for October, Autumn, Winter, and Spring, which we average across all grid cells that are within 50 km of the Los Pedroches region for each year of the analysis. This is the same method used for the observations and produces the forecast values for the contingency table evaluation. For the mean, upper tercile and lower tercile of the ensemble, we generally find that the effect of the bias correction is to increase precipitation values compared to SEAS5, as can be seen in the example forecasts in (Figure 38). 56 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 38 Ensemble forecast means for rainfall in spring compared to eventual observed values in 2013 and 2014. Ideal values and adequate ranges are also provided. Note the effect of bias correction (bottom) is to increase mean rainfall relative to the raw SEAS5 57 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.3.1.3 Forecast evaluation Interpolated precipitation is compared to both SEAS5 and SMHI bias corrected forecasts using a contingency table approach. For each period (October, Autumn, Winter, and Spring) we categorise the area averaged accumulated precipitation in each year as ‘adequate’, ‘less than adequate’ or ‘more than adequate’ according to the ‘adequate’ ranges in Table 12. In total, 30 years are analysed for October, Autumn, and Spring. 29 years are available for Winter because we do not have observation data from December 1992. Table 12 Farmers’ ideal precipitation and the ‘adequate’ ranges defined in this study. *For spring we adjusted the Dehesa farmers values to those in brackets as 400 mm was outside the observed range of spring values in Los Pedroches. Time period Farmers’ Ideal Precipitation Value(s) (mm) ‘Adequate’ Precipitation Range used in forecast evaluation (mm) October 100 50-150 Autumn (SON) 150-200 100-300 Winter (DJF) 200-300 125-375 Spring (MAM) 400 (200)* 200-600 (100-300)* We evaluate all forecasts that contain the whole period of interest (i.e. October, Autumn, Winter or Spring). For example, the observed precipitation for Autumn (SON) is compared to accumulated SON precipitation predicted by forecasts run on 01 May, June, July, August and September (1-5 month lead time). For Spring 1993, we only analyse 29 forecasts at 4and 5months lead times, because forecasts from November and December 1992 are not available. For every forecast, each ensemble member is categorised as ‘adequate’, ‘less than adequate’ or ‘more than adequate’ for each accumulation period and year of the record. We then calculate the overall probability of being in a given category by dividing the number of members by the total number of members (26 or 51 depending on date) * 100 (Figure 39; although note this shows fraction rather than probability). We generally find that the effect of bias correction is to increase the probability of rainfall being in ‘adequate’ and ‘more than adequate’ categories, and decrease the probability of being in the ‘less than adequate’ category (Figure 39). This is in line with the typical increase in absolute precipitation amounts. 58 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 39 Example forecast probabilities of ‘adequate’, ‘less than adequate’ and ‘more than ‘adequate’ rainfall for Spring 2013 and 2014 from SEAS5 and SMHI bias corrected SEAS5. The equivalent observed precipitation in each year is shown for comparison. These are the probabilities for the same forecasts as shown in Figure 38, but note that that figure shows the ensemble mean, whilst this shows the probability of being in each of our three categories. We evaluate forecast performance for predicting each of the three categories using the contingency table (Figure 4) as follows: For each ensemble probability between 10 and 100, in 10% increments, a hit is scored if forecast and observation are in the same category, a miss is scored if the observation is in the category but the forecast is not, a false alarm is scored if the observation is not in the category but the forecast is, and finally a correct rejection is scored if observation and forecast are both not in the category. For example, if the ensemble probability of ‘adequate’ precipitation is 30% or above, and the observations are also in the ‘adequate’ category, the ensemble scores a hit at the 30% level, and so on. Finally, we sum up the hits/misses/false alarms/correct rejects at each lead time to get an overall performance assessment in each precipitation category. We use these to calculate frequency bias and threat score (Table 1). 59 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.4 Results 4.4.1 Historical precipitation Before evaluating the forecast data, we first consider the Farmers’ ideal precipitation values, and our ‘adequate’ ranges, in the context of the historical records. In the timelines exercise, the Farmers provided an indication of good and bad years, and ideal precipitation for the different accumulation periods. We computed the observed precipitation in each year for each accumulation period. These values across all the 30 years were used to compute the accumulated precipitation associated with the median, upper and lower terciles. Results were compared to the values associated with ideal conditions provided by the Farmers. It might be expected that these ideal conditions would lie close to the median value defined by the observed precipitation, and that these ideal conditions should be met relative frequently during the observation period. The Farmers’ ideal annual precipitation values ranged from 300 mm to 600 mm (dashed lines in Figure 40), and these conditions were met in 16 out of 30 years. Figure 40 Hydrological year historical rainfall from interpolated observations 1993-2022. Dashed lines enclose the range of values that farmers provided as moderate to good in the timelines exercise. 2020 was highlighted as a good year by Dairy and Olive farmers (the Olive farmers noted that this refers to the hydrological year running from October 2019 to September 2020). For the Dairy farmers, this was due to overall good levels in aquifers, although they stated that it was not one of the years with the most rainfall. The Olive farmers stated that the 2020 hydrological year saw 457 mm rainfall, with good rain in Autumn and Spring. Our interpolated rainfall observation data yield a value of approximately 493 mm for 2020, which agrees well with the value given by the farmers. 1996 was noted as a bad year by the Dairy farmers due to a lack of fodder production. Whilst the hydrological year of 1996 shows very high rainfall in our record (Figure 40), winter fodder crops would have been planted in the October of 1995, which was the driest October in our record with only 6 mm of rain (Figure 41). Rainfall records for October, Autumn, Winter and Spring, are shown in Figure 41 to Figure 44 along with the median, upper and lower terciles. For October (Figure 41) the Farmers’ ideal value of 100 mm is significantly above the upper tercile of the observations (76 mm). There are 15 ‘adequate’ years (100-300 mm), 12 ‘less than adequate’ years, and 3 ‘more than adequate’ years. 60 D3.4 User-driven Evaluation of the I-CISK Climate Services ‘Less than adequate’ October rainfall (12 observed years; top plots in Figure 47): Frequency bias typically approaches one (a perfect score) at probabilities of 60-70% for SEAS5 and 40% for the SMHI bias corrected, with a tendency toward higher probability at the shortest lead times (e.g. p=50% for the SMHI bias corrected at 1 month lead time). At lower probabilities, both forecasts rapidly start to predict ‘less than adequate’ rainfall all the time. The ‘topping out’ value of 2.5 (threat score of 0.4) is reached p=30-40% for SEAS5, and 20% the SMHI bias corrected at lead times of 2-7 months. The lower probabilities for SMHI bias corrected, as compared to SEAS5, show that the effect of bias correction is to decrease forecast frequency of ‘less than adequate’ October rainfall. The maximum threat score attained is 0.44 for SEAS5 (1 month lead, p=40% and 70%) and 0.47 for SMHI bias corrected (1 month lead, p=50% and 2 months lead, p=40%), so at best less than half of ‘less than adequate’ forecasts are correct. A very slight improvement due to bias correction is seen both in the maximum threat score and in the range of threat scores at probability levels where frequency bias is typically closest to one (0.29-0.4 at 60% for SEAS5 and 0.23-0.47 at 40% for the SMHI bias corrected). However, considering as an example the raw hit, miss, and false alarm counts for the maximum threat scores, we can see that the difference in performance is negligible, equating to just one less false alarm in the SMHI bias corrected forecast (Figure 48 h=8, m=4, fa=6 for SEAS 5 at 1 month, p=70% and h=8, m=4, fa=5 at 1 month, p=50% for SMHI bias corrected). Figure 48 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Less than adequate’ rainfall in October. SEAS5 hit counts tend to increase toward shorter lead times at p=50, 60 and 70%, (although there is curious drop at 1 month for both p=50 and 60%), whilst for SMHI this is only really apparent at p=50% (Figure 48). A decrease in false alarms at shorter lead times is only evident from 3-4 months lead time at p= 50% (SEAS5) and 30% (SMHI bias corrected). ‘Adequate’ October rainfall (15 observed years; middle plots in Figure 47): In this category, frequency bias typically approaches one at p~30-40% for SEAS5 and ~50% for SMHI bias corrected. Again, at lower probabilities both forecasts trend toward predicting ‘adequate’ rainfall all the time. SEAS5 reaches the ‘topping out’ value of 2 (threat score of 0.5) at p=10% and SMHI bias corrected between p=20 and 40%, 67 D3.4 User-driven Evaluation of the I-CISK Climate Services depending on lead time. The lower probabilities for SEAS5 suggest the effect of bias correction is to increase the forecast frequency of ‘adequate’ October rainfall, opposite to what is observed for the ‘less than adequate’ category. The maximum threat score attained is 0.52 for SEAS5 (4 and 6 months lead, p=20%,) and 0.54 for SMHI bias corrected (2 month lead, p=40%), indicating only a little more than half of ‘adequate’ forecast are correct. The very slight improvement due to bias correction is also seen if we compare the threat score range at probability levels where frequency bias is typically closest to one (0.22-0.44 at 30-40% for SEAS5 and 0.230.46 at 50% for the SMHI bias corrected). As for the ‘less than adequate’ category, the difference in maximum threat scores between the two forecasts equates only to one less false alarm for the SMHI bias corrected forecast (Figure 49: h=15, m=0, fa=14 for SEAS 5 at 4 and 6 month, p=20% and h=15, m=0, fa=13 at 2 month, p=40% for SMHI bias corrected). The raw hit/miss/false alarm counts also reveal that in this category the highest threat scores simply result from over-forecasting ‘adequate’ rainfall amounts, which increases the number of hits and false alarms. Figure 49 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Adequate’ rainfall in October. None of the probability levels shown in Figure 49 display a systematic increase in hit count with shortening lead time, whilst at p=30% for SEAS5 and p=50% for SMHI bias corrected, false alarm counts do generally decrease. Overall, however, threat scores do not particularly improve for either forecast at shorter lead times. ‘More than adequate’ October rainfall (3 observed years; bottom plots in Figure 47): No SEAS5 forecast between 1993 and 2022 predicted more than 150 mm of rainfall in October. The SMHI bias corrected did, with frequency bias nearing 1 at about p=20%. When combined with the observed increase in instances of ‘adequate’ rainfall and deceased instances of ‘less than adequate’ rainfall from the SMHI bias corrected forecast relative to SEAS5, this suggests bias correction increases forecast October rainfall in Los Pedroches. 68 D3.4 User-driven Evaluation of the I-CISK Climate Services Given there are very few observed years with ‘more than adequate’ rainfall, it is difficult to assess model performance in this category (for reference, hit, miss and false alarms counts for the SMHI bias corrected forecast are shown in Figure 50). Figure 50 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. Scores for forecasting ‘More than adequate’ rainfall in October. Only SMHI evaluation is shown because SEAS5 forecasts no events 69 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.4.2.2 Autumn (‘Adequate’ range = 100-300 mm) Figure 51 Frequency bias (left) and Threat score (right) for each category of rainfall (‘less than adequate’; top, ‘adequate’; middle, and ‘more than adequate’; bottom) in Autumn. In each plot, SEAS5 results are shown on the left and SMHI bias corrected on the right. The zone of ‘topping out’ i.e. forecasting a given category all of the time, is delineated using the thick, dark blue line. Maximum threat score for each category/forecast is highlighted using a red box. 70 D3.4 User-driven Evaluation of the I-CISK Climate Services ‘Less than adequate’ Autumn rainfall (5 observed years; top plots in Figure 51): Frequency bias typically approaches one at probabilities of 50-60% for SEAS5 and 30% for SMHI bias corrected. The ‘topping out’ value of 6 (threat score=0.17) is reached by the SEAS5 forecast at probabilities of 10-20% and is not observed for the SMHI bias corrected. As is the case for October, then, bias correction decreases the forecast frequency of ‘less than adequate’ rainfall. Figure 52 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Less than adequate’ rainfall in Autumn. The SEAS5 maximum threat score is 0.29 at p=40% (1 month lead time: h=4, m=1, fa=9, and 4 month lead time: h=5, m=0, fa=12, Figure 52) and at p=60 (4 month lead time: h=2, m=3, fa=2). The maximum score for SMHI bias corrected is higher, at 0.43 (h=3, m=2, fa=2: 5 month lead time, p=30%), but when compared to the p=60%, 4 months lead time value for SEAS5, the scores are only different by one hit transitioning to a miss. None of the probability levels in Figure 52, for either forecast, show an increase in hits or decrease in false alarms as lead time shortens. It’s difficult to draw robust conclusions in this category due to the limited number of historical occurrences of ‘Less than adequate’ Autumn rainfall. ‘Adequate’ Autumn rainfall (23 observed years; middle plots in Figure 51): Frequency bias typically approaches one at p~50-60% for SEAS5 and ~70% for SMHI bias corrected. SEAS5 reaches the ‘topping out’ value of 1.3 (threat score of 0.77) at p=20-30%, whilst for the SMHI bias corrected this occurs at p=40 and 50%, depending on lead time. The lower probabilities for SEAS5 suggest that bias correction increases the forecast frequency of ‘adequate’ Autumn rainfall, this is the opposite of what is observed for the ‘less than adequate’ category. This trend is the same as that observed for October, where reduced occurrences of ‘less than adequate’ rainfall, and increased occurrences of ‘adequate’ rainfall are also observed for the bias corrected forecasts relative to raw SEAS5. Again taken over this shows the effect of bias correction is typically to increase precipitation amounts. 71 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 53 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Adequate’ rainfall in Autumn. The maximum threat score for both forecasts is 0.82 (h=23, m=0, fa=5 in both instances, Figure 53), occurring at p=40%/4 months lead time for SEAS5 and p=60/5 months lead time for SMHI bias corrected. From the raw hit, miss, false alarm counts (Figure 53), this high score does not appear to reflect forecast skill, but rather over-forecasting of a frequently occurring event (‘adequate’ rainfall). Indeed, Figure 53 shows that as soon as false alarm counts drop with increasing probability, hit counts also rapidly decline. If we compare the 50-60% (SEAS5) and 70% (SMHI bias corrected) probabilities, where frequency bias is generally close to 1, the threat score ranges are 0.31-0.72 and 0.55-0.64 respectively. Neither in these ranges, nor in the maximum threat scores, does there seem to be an improvement in forecast skill resulting from bias correction. There is no systematic increase in hits/decrease in false alarms with shortening lead time (Figure 53). Therefore, despite the deceptive high threat scores, neither model is particularly skilful at predicting ‘Adequate’ Autumn precipitation in Los Pedroches. ‘More than adequate’ Autumn rainfall (2 observed years; bottom plots in Figure 51): As for October, there are very few observed years with ‘more than adequate’ rainfall in Autumn, making it difficult to assess model performance. The same pattern is observed for Autumn as for October, with no SEAS5 forecast predicting rainfall in the ‘more than adequate’ category. SMHI bias corrected does predict some instances, with frequency bias nearest 1 at about 20%. Again, this indicates that the effect of the bias correction is to increase forecast rainfall amounts. There is a hint of skill from the bias corrected forecast in this category, with maximum threat scores of 0.5. However, given the small number of events, drawing robust conclusions is not possible. 72 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.4.2.3 Winter (‘Adequate’ range = 125-375 mm) Figure 54 Frequency bias (left) and Threat score (right) for each category of rainfall (‘less than adequate’; top, ‘adequate’; middle, and ‘more than adequate’; bottom) in Winter. In each plot, SEAS5 results are shown on the left and SMHI bias corrected on the right. The zone of ‘topping out’ i.e. forecasting a given category all of the time, is delineated using the thick, dark blue line. Maximum threat score for each category/forecast is highlighted using a red box. 73 D3.4 User-driven Evaluation of the I-CISK Climate Services ‘Less than adequate’ Winter rainfall (13 observed years; top plots in Figure 54): Frequency bias is generally closest to one at p=40% for SEAS5 and 30% for SMHI bias corrected, although both have a tendency for values near one to extend toward higher probability at the shortest lead times (e.g. bias is 0.92 at p=50% for SEAS5 at 1 month lead time). Both forecasts predict ‘less than adequate’ rainfall all the time at p=10% for lead times of 2-5 months (frequency bias of 2.23 and threat score of 0.45). The tendency of bias correction to decrease the occurrence of ‘less than adequate’ rainfall is weak in Winter, with only a subtle difference in the probability where threat scores near 1/ ‘topping out’ values are observed. From our data it is not clear whether this is because of a smaller correction or because the bottom of the ‘adequate’ range is higher (i.e. the category is less extreme) than in other periods considered so far. Figure 55 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Less than adequate’ rainfall in Winter. The SEAS5 maximum threat score is 0.68 (1 month lead, p=40%: h=13, m=0, fa=6, Figure 55), with 0.59 for SMHI bias corrected (1 month lead, p=30%:h=10, m=3, fa=4). SEAS5 does seems to show some forecast skill, particularly at the shortest lead times and it is encouraging to note it displays the expected increase in hits toward shorter lead times at p=40% (Figure 55). A decrease in false alarms is evident from 3 months lead time; this yields threat scores that actually increase with shortening lead time across this probability level. There appears to be no benefit gained by bias correction in this category, either for the maximum threat scores achieve or for the range of threat scores observed at probability levels where frequency bias is closest to one (0.25-0.68 at p=40% for SEAS5 and 0.15-0.59 at p=30% for SMHI bias corrected). ‘Adequate’ Winter rainfall (13 observed years; middle plots in Figure 54): Frequency bias is closest to one at p= 60% for SEAS5 and ~70% for SMHI bias corrected. SEAS5 reaches the ‘topping out’ value of 2.23 (threat score of 0.45) at p=30-40% and SMHI bias corrected at p=30-50%, depending on lead time. The probabilities values are only slightly high for the SMHI bias corrected vs. SEAS5, indicating a slight increase in the forecast frequency of ‘adequate’ Winter rainfall. 74 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 56 Raw counts of hit, miss, false alarm and correct rejection for probability levels proximal to frequency bias approaching 1. SEAS5 top, SMHI bottom. Scores for forecasting ‘Adequate’ rainfall in Winter. The maximum threat score attained is 0.5 for SEAS5 at 1 month (h=12 m=1 , fa=11 at p=40%, and h=10, m=3, fa=7 at p=50%,Figure 56) and 4 months lead time (h=13 m=0 , fa=13, at p=50%). For the SMHI bias corrected forecast, a maximum of 0.52 is attained at 4 months lead time and p=60%, (h=12 , m= 1, fa= 10). Overall, there is little difference in maximum score between the raw and bias corrected forecasts, which is also reflected in the threat score range at probability levels where frequency bias is typically closest to one (0.29-0.42 at 60% for SEAS5 and 0.15-0.39 at 70% for the SMHI bias corrected). Here again then, no more than half of forecast events are correct (as a maximum) for either forecast. This seeming lack of skill is reflected in the absence of a clear trend to increasing hits, decreasing false alarms, or increasing threat score as lead time shortens (Figure 56). ‘More than adequate’ Winter rainfall (3 observed years; bottom plots in Figure 54): Again, there are few observations in this category so drawing conclusions on forecast performance is challenging. SEAS5 never forecasts winter rainfall in the ‘more than adequate’ category, whilst the SMHI bias corrected only does so a handful of time (e.g. 3 instances at p=10% and 4 months lead time). The increase in number of forecasts in this category resulting from bias correction is less than for October and Autumn, which, as for the ‘less than adequate’ and ‘adequate’ categories, suggests that either the forecast correction in Winter is less or the ‘adequate’ range investigated is different. 75 D3.4 User-driven Evaluation of the I-CISK Climate Services 4.4.2.4 Spring (‘Adequate’ range = 100-300 mm) Figure 57 Frequency bias (left) and Threat score (right) for each category of rainfall (‘less than adequate’; top, ‘adequate’; middle, and ‘more than adequate’; bottom) in Spring. In each plot, SEAS5 results are shown on the left and SMHI bias corrected on the right. The zone of ‘topping out’ i.e. forecasting a given category all of the time, is delineated using the thick, dark blue line. Maximum threat score for each category/forecast is highlighted using a red box. 76 D3.4 User-driven Evaluation of the I-CISK Climate Services 5 Emilia Romagna Living Lab (Italy) 5.1 Summary Seasonal forecasts of streamflow to identify periods of low flows in the Secchia river within the Emilia Romagna living lab were evaluated utilising thresholds provided by the stakeholder. Initial evaluation found very low forecast skill at all lead times, with the optimum threat score being close to 0.20, this was due to a large number of missed events. The evaluation was re-performed using a method recently applied in CEMS-Floods to identify whether forecasted streamflows are lower than, close to, or above normal conditions. Seasonal reforecast climatology data were used to define the normal streamflow conditions. However, the evaluation scores using this new method did not yield any improvements, the threat score remained close to 0.20 due to many missed events. Therefore, the seasonal streamflow forecasts in the Secchia river are not yet suitable for the stakeholder to anticipate low streamflows. 5.2 Introduction The Italian living lab is focussed upon the impacts of low river flow conditions upon the Secchia river, a tributary of the Po River, located in the region of Emilia-Romagna in northern Italy (Figure 60). Low flow conditions are defined when the streamflow is lower than a Minimum Environmental Flow (MEF), which describes the minimum streamflow to sustain the ecology of the river. The stakeholder in the living lab would like to use climate services to forecast low flow conditions over the next six months. Figure 60 Location of the Secchia river basin and the two river gauging stations where observations of low flow conditions were available. 5.3 Methodology 5.3.1 Observation Data Observations of daily average streamflow were available at two gauging stations within the living lab study area: 1) Ponte Veggia where data were available between 25/06/2021 to 31/10/23, and 2) Lugo where data 83 D3.4 User-driven Evaluation of the I-CISK Climate Services were available from 01/01/2003 to 28/08/2024. These data were provided by the regional environment agency – ARPAE Emilia-Romagna. Both stations were located on the mainstem of the Secchia river, the Lugo station was located furthest upstream, the Ponte Veggia station was located 17 km further downstream near the town of Sassuolo. There are two significant artificial structures close to these gauging stations which may alter river flow conditions. The first is the Castallerano dam, located 6 km upstream of Ponte Veggia, whose primary purpose is electricity generation. The second structure, located at Ponte Veggia, is a barrage whose primary purpose may be flood prevention. It was decided that only data from the Lugo gauging station would be used in the analysis of the climate service data. This is because its observations were unaffected by the artificial structures within the catchment and it has a longer observation period. Each streamflow observation was classified into one of three low flow categories: green, yellow or red, where red was the most severe low flow category. These were defined using criteria defined by ARPAE Emilia-Romagna which are based on the MEF (Minimum Environmental Flow) threshold at the Lugo station (Table 16). Typically, the MEF is defined as 10% of the mean annual observed river flow, but the precise methodology used to compute the threshold at this location was not provided. The MEF represents the minimum streamflow required to sustain the ecology of the river and any associated human livelihoods. Table 16 Minimum Environmental Flow (MEF) threshold and the low flow categories and their criteria at the Lugo gauging station. Station Name MEF Threshold (m3 s-1) Lugo 1.77 Low Flow Category Criterion Green River flow > (MEF +30%) Yellow (MEF - 30%) < River flow < (MEF + 30%) Red River flow < (MEF – 30%) The above low flow category criteria were applied to historical observation time series of streamflow at the Lugo station. The total number of days in each year when the yellow and red low flow categories were calculated. The results show that yellow and red low flow categories occurred in all years of the observation period with the exception of 2013, 2014 and 2020 (Figure 61). The yellow low flow category occurred more frequently than the red low flow category. However, 2017 and 2023 the reverse was true, suggesting that low flows during this year were severe. Observations since 2017 show a higher number of days with either yellow or red low flow categories than years prior to this, it is unclear if this trend reflects a broader climatological trend. 84 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 61 Number of days during each year when the observed streamflow at the Lugo gauging station (between 20032023) was classified as either the yellow or red low flow category. To analyse the seasonal distribution of low flows, the above analysis was repeated to count the total number of days in each month when yellow or red low flow categories were observed. Results show the highest frequency occurred between July to October (Figure 62), which coincides with the summer period when prolonged periods of drought can occur. Red low flow conditions were also observed between November to January, which is surprising as typically the winter period is associated with wetter conditions. However, drought events have been observed during winter, for example in January 2019. It is unlikely that these winter droughts could have been caused by the accumulation of a large snowpack because hydrological processes associated with snow play a limited role within the Secchia river catchment. Figure 62 Number of days during each month when the observed streamflow category at the Lugo gauging station (between 2003-2023) was classified as either the yellow or the red low flow category. 85 D3.4 User-driven Evaluation of the I-CISK Climate Services 5.3.2 Forecast Data Seasonal forecast data of river discharge were available from the e-HYPE (european-HYdrological Predictions for the Environment) hydrological model at SMHI (Sveriges Meteorologiska och Hydrologiska Institut – Swedish Meteorological and Hydrological Institute). The model has a lumped structure meaning it calculates hydrological fluxes across sub-catchments. Evaluation of the hydrological performance of the e-HYPE model, when forced with daily meteorological re-analysis data from ERA5, found moderate performance at the Lugo station (Table 17), as shown by a KGE score of 0.51 relative to a perfect score of 1.0. The NSE score suggests a poor performance, but this metric gives more weight to the prediction of high streamflows, which are less relevant for this living lab which is focussed on the prediction of low streamflows. The results of the hydrological performance will provide context when evaluating the ability of the e-HYPE seasonal streamflow forecasts to predict low streamflow events. Table 17 Hydrological performance metrics of the e-HYPE model when forced with meteorological re-analysis data against streamflow observations at the Lugo station. Performance Metric Score Modified Kling-Gupta Efficiency (KGE) 0.51 (Correlation = 0.62, variability = 0.71, bias = 0.92) Nash Sutcliffe Efficiency (NSE) 0.37 Mean Absolute Error (MAE) 9.97 Daily river discharge forecasts were generated on the first of each month between 1st January 1993 – 1st December 2022 using ensemble meteorological forcings from the SEAS5 NWP seasonal re-forecasts generated at ECMWF. Reforecasts generated before 2017 had 25 ensemble members, reforecasts thereafter had 51 members. The river discharge reforecasts were available at the outlets of each of the subcatchments shown in Figure 63. The discharge reforecasts produced at the outlet of subcatchment ID 9780124 were used in the analysis, data from the same subcatchment were used to compute the hydrological performance (Table 17). 86 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 63 Locations of the sub-catchments of the e-HYPE hydrological model used to generate the streamflow forecasts in the Secchia river. 5.4 Results 5.4.1 Evaluation of the Reforecast Data A first analysis evaluated the ability of the reforecast data, aggregated to weekly averages, to predict low flow conditions using the thresholds provided by the stakeholders (Table 16). For each ensemble member of a reforecast, the daily river discharge reforecast data in each ensemble member were aggregated to weekly values, defined as the average over fixed seven-day periods, rather than seven day moving averages. This was applied up to a maximum lead time of 30 weeks. The observed river discharge data were also aggregated to compute the weekly average across the same fixed seven-day periods for which the reforecast data were aggregated. The threshold associated with yellow low flow category conditions were then applied to both the weekly averaged observed and reforecast data. At each time step in the reforecast, the percentage of ensemble members which were predicted as having yellow low flow category conditions was computed. The steps described in this paragraph were repeated for each reforecast produced on the first of each month. The weekly averaged reforecast data were then evaluated against the weekly averaged observed data. At each weekly timestep of the reforecast, different thresholds, pertaining to the percentage of ensemble members predicting yellow low flow conditions, were applied to generate yes/no forecasts of yellow low flow conditions. These yes/no forecasts were evaluated against the observed yellow low flow category status using a contingency table, which recorded the number of hits, misses and false alarms. Separate contingency tables were computed for each weekly lead time and each forecast percentage threshold of yellow low flow conditions. The results in each contingency table were used to compute the Threat score (also known as the Critical Success Index), which measures the fraction of the correctly forecasted events relative to missed events and false alarms. 87 D3.4 User-driven Evaluation of the I-CISK Climate Services Results from the analysis of the Threat score show very low skill across all lead times and percentage thresholds (Figure 64). Beyond a lead time of 14 weeks, the Threat score was 0.00 for all percentage thresholds apart from 10% and 20%. The highest scores, with a maximum value of 0.21, were obtained when using a 10% probability threshold of yellow MEF conditions. Figure 64 Threat score values obtained when applying different thresholds of % of forecast ensemble members below the stakeholder defined yellow MEF low flow conditions and evaluated against observed low flow conditions between 1/1/2003-31/12/2022 To investigate the results in Figure 64 further, the number of hits, misses and false alarms, which constitute the Threat score, were plotted for all lead times for percentage thresholds of 10, 20, 30 and 50% (Figure 65). The results for all four thresholds are dominated by many missed events. Results from the 10% percentage threshold give the largest number of hits, which explains the higher Threat score values, but this is accompanied with a larger number of false alarms. These results explain that the low Threat score values are the result of a high number of missed events, this could be because the forecasted river discharge rarely falls below the threshold associated with yellow MEF low flow conditions, possibly suggesting that the e-HYPE forecasts in the Secchia river are positively biased. The results from the original hydrological evaluation (Table 17) do not support this, as the bias value of 0.92 suggests a slight negative bias. To investigate further, the bias was computed using data only for the months of July to October, which correspond to the main low flow season. Daily river discharge forecast data from the control ensemble member were extracted from each monthly forecast for lead times up to four weeks and were evaluated against the daily observed river discharge. The bias during the low flow season was 1.82, suggesting a positive bias which could explain the large number of misses when predicting yellow MEF low flow conditions. The results suggest that applying a low flow threshold, derived from observed data, to positively biased forecast data will result in poor predictive skill. 88 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 65 Hits, misses and false alarms obtained when low flow events are defined when 10%, 20%, 30% and 50% of the forecast ensemble members are below the stakeholder defined yellow MEF low flow conditions. 5.4.2 Evaluation against Warnings Derived from Model Reforecast Climatology An alternative method, using climatological percentiles derived from the 30 year period of model reforecasts, was used to predict low flow events. Deriving the percentiles directly from the model reforecasts circumvents the issue of model bias, as the percentiles will have the same bias as the forecasted values. This alternative method was recently developed at ECMWF (ECMWF, 2025) and has been applied to the EFAS seasonal and sub-seasonal forecasts since the release of version 5.3 in March 2025. The climatological percentiles are computed separately for each month and each weekly lead time within the monthly reforecasts. For example, to compute the percentiles for January, only the reforecasts created in January during the 30-year reforecast period were used. For each January reforecast, the weekly average streamflow was computed in each ensemble member up to a maximum lead time of 30 weeks. Next, for each weekly lead time, the weekly average streamflow from all ensemble members and all years were extracted and used to compute the streamflow associated with percentiles from 0 to 100. This procedure is repeated for all weekly lead times and for each month. The percentiles, computed above, are combined with forecasted weekly average streamflow to assign one of seven flow categories, ranging from extreme low to extreme high. Firstly, for a given forecast, the weekly average forecasted streamflow was computed for all ensemble members. These were converted into percentiles using the climatological percentiles for the relevant month and weekly lead time. The streamflow category was computed using rules based on the ensemble mean of the percentiles (Figure 66). For example, a forecast would be assigned the ‘extreme low’ category if the mean percentile was less than or equal to 10%. 89 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 66 Rules used to define streamflow categories, note that 'Rank' refers to the ensemble mean percentile The methodology described above was applied to the seasonal streamflow forecasts in the Secchia river to define low flow events at the Lugo gauging station. Low flow events were defined from the forecast by defining thresholds related to the first of the four streamflow categories: 1. Extreme low, 2. Low, 3. Bit low and 4. Normal (Figure 66). A forecasted low flow event was defined if the forecasted streamflow category was equal to or less than the threshold category. These were evaluated against the observed occurrences of low flow conditions associated with the yellow MEF threshold, using the same methodology described previously. Results from the evaluation when using the streamflow categories defined from forecast climatological percentiles showed higher skill (Figure 67) than the results obtained when applying the stakeholder defined threshold to define forecasted low flow events. However, the computed Threat scores were still very low, the maximum value was 0.25 compared to an optimum value of 1.00. Defining low flow events when the forecasted streamflow category was <=1 (Extreme low only) produced very poor skill, with a maximum value of 0.11 at 1 week lead time. Better skill was obtained when defining low flow events from forecasted streamflow categories <=3, but only up to a lead time of 5 weeks. The most consistent results across all lead times were when low flow events were defined from forecasted streamflow categories <=4. However, this particular criterion would refer to streamflow conditions being within a broad percentile range of 0-60%, which could lead to an over-warning of low flow conditions if this criterion is used. 90 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 67 Threat score values obtained when applying different thresholds of forecasted streamflow category and evaluated against observed low flow conditions associated with the stakeholder defined yellow MEF criterion between 1/1/2003-31/12/2022 To investigate further, the number of hits, misses and false alarms were visualised for each of the four thresholds (Figure 68). The results confirm that defining low flow events when the forecasted streamflow category is <=4 results in a large number of false alarms. Results when defining low events when the forecast streamflow categories are <=1 and <=2 also show poor performance due to the very low number of hits. Defining low flow events when the forecasted streamflow category is <=3 gives the best results, with more hits than were achieved from the first two categories and fewer false alarms than obtained from the fourth category. This streamflow category gave the best results during the first three weeks of lead time, afterwards there is some skill out to nine weeks lead time but beyond this lead time there are only a minimal number of hits. 91 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 68 Hits, misses and false alarms obtained when defining low flow events using four different streamflow category thresholds and evaluated against observed low flow events defined using the stakeholder defined yellow MEF criterion between 1/1/2003-31/12/202 The above evaluation was repeated to investigate if higher skills were achieved when predicting the more severe low flow category associated with the red MEF criterion. The results however showed minimal skill, with a maximum Threat score of 0.14, this was due to a very low number of successfully forecasted events, i.e. hits. The poor results above suggest that the forecasts cannot predict severe low flow events, to investigate further, the forecasted streamflow categories ahead of and during the major drought event of 2022 were plotted. The 2022 drought in the Po basin was driven by moderately lower than normal rainfall in combination with a reduced snowpack accumulated during the winter (Montanari et al. 2023 ). Longer term climatic trends including an increase in evaporation due to rising temperatures and the expansion of irrigated areas also contributed to the severity of the drought. In the Secchia river, the drought would likely have been driven by the rainfall anomaly and higher evaporation due to the lower importance of snowmelt and irrigation upon the hydrology of the catchment. 92 D3.4 User-driven Evaluation of the I-CISK Climate Services Jan Feb Mar Apr May Jun July Aug Sep Oct Nov Dec Climate Characteristics Winter (cool & mild) & Weather observations Rain with intermittent dry spells High Temperatures & Low precipitation Rain with intermittent dry spells Sectoral activities Trimming and Pruning Vines Pesticide Operations Harvest Wine Production Climate related risks Frost & excessive precipitation Water shortages & high temperatures Hail & intense precipitation Too much precipitation Socio-economic factors Availability of human resource (labour) Availability of human resource (labour) & machinery Environmental cues Movement of the clouds (direction of movement as a precursor of weather conditions) Socio-economic cues Consultation with other farmers & farmer associations Demand from wine companies CS & information sources Google weather, yr.no (relying on weekly forecasts) and various websites offering farming advice Coping Strategies Delay trimming and tying up of vines Use water from irrigation or wells Adaptation Strateges Invest in drip irrigation systems Combine wine making with agro-toursim Spring (transition) Summer (hot & humid) Autumn (transition) Figure 73 Decision timeline for wine farmers in the Alazani-Iori Living Lab. 99 D3.4 User-driven Evaluation of the I-CISK Climate Services The co-evaluation framework and results presented in this section followed the work of Correa (2024) and aimed to assess whether the forecasts aligned with decision-making processes, were useful, and provided timely drought alerts to support early warning and planning. The co-evaluation session involved tailored visualisations and a structured questionnaire covering six key aspects: self-assessment, utility, understandability, coherence, credibility, and engagement. The idea behind was to ensure that both the technical performance and practical usefulness of the forecasts were evaluated. 6.3.2 Forecast evaluation and user feedback For this evaluation we used streamflow observations from the Shaqriani station and seasonal reforecasts obtained from the CEMS-Floods EFAS seasonal streamflow forecasts at the same location. The Shaqriani station was used due to its long observation record which aligned with the EFAS reforecast period. However, inspecting the observed time series, we identified unusual values in different years—such as 1996, 2000, 2001, 2013, and 2018—that suggested possible measurement errors or data gaps (Figure 74). During a visit to the Shaqriani station it was noted that such higher flows during low flow periods would be improbable for that region. Due to these differences, the determination of verification with metrics is based on relative skill against EFAS streamflow predictions forced with meteorological observations (subsequently referred to as reanalysis). This comparison limits the impact of model bias, uncertainty, and potential measurement errors in the observed data. Figure 74 Comparison of observed discharge and EFAS historical reanalysis at Shaqriani Station. Despite the differences depicted in Figure 74, the comparison of cumulative distribution functions for low flows (Figure 75) shows that the probability of non-exceedance between the observed and reanalysis datasets for low flows is not significantly different, supporting the use of either for threshold definition. 6.4 Results Following the approach described above, we derived discharge thresholds based on statistical percentiles for the summer season (June–August), focusing on the lower range of the flow distribution. As shown in the comparison of cumulative distribution functions (Figure 75), the 10th percentile of discharge was 100 D3.4 User-driven Evaluation of the I-CISK Climate Services approximately 9 m³/s, the 20th percentile around 10 m³/s, and the 33rd percentile around 12/13 m³/s in both the observed and reanalysis datasets. Figure 75 Proposition of threshold definition and alert levels – comparing using observations and historical reanalysis. Figure 76 shows the pattern of low flows in this part of Georgia, which is influenced by the hydrology, seasonal variations and snowmelt dynamics present in the region, as fast responding convective events in summer. During winter months (January to March), the flows are relatively low due to less precipitation and lower temperatures, which leads to reduced runoff and accumulation of snow. The spring months (April and May) present an increase in discharge due to snowmelt from the mountains, contributing to higher flows. However, during the summer months (June to August), even with some snowmelt, the overall flows decrease as precipitation reduces and temperatures rise, which leads to higher evaporation rates. Even though July and August are not the months with the lowest minimum average discharge, the thresholds are proposed for these months due to their decision-making relevance. Previous investigations have identified that these months are crucial for water allocations to irrigated agriculture. Setting thresholds for June, July and August is important for managing water resources effectively during periods when agricultural demand is higher for example. Figure 76 Average minimum discharge from EFAS reanalysis compared with monthly thresholds. 101 D3.4 User-driven Evaluation of the I-CISK Climate Services The verification period comprised years from 1991 to 2018, resulting in 81 available pairs (one pair is a single forecast at a particular lead time and a corresponding observation) for the months of interest (June, July, August). However, when applying thresholds, the number of available pairs decreases. For instance, at a threshold of 12 m³/s and a 7-month lead time, there are 53 pairs available and for a 1-month lead time, there are 5 pairs available. This fact implies an increase of uncertainty in the computed metrics. For clarity in the visualizations, the threshold flow of 12 m³/s is used for display purposes. Given the additional interest in monthly aggregated discharges for water allocation and hydropower operations in Georgia, the data is aggregated to show the monthly behaviour for the “eyeball” verification. Even if the observed data is not being used for the verification metrics, it is kept in the visual inspection graphs. Taking the 2018 event as an example (Figure 77), observed data reveals higher discharge values than model results from May to July, likely due to late snowmelt that the model fails to capture. During these months, observed discharges show higher peaks and more persistent flows. From August to December, discharge values decrease significantly and align more closely with model predictions, indicating better model performance in capturing lower flows and baseflow conditions. Figure 77 Visual inspection of EFAS forecasts from 2018 drought event for months of interest in terms of daily discharge. Forecast initiation time is in the title of each subplot. Additionally, looking at the same event but with monthly aggregated discharges (Figure 78) the boxplots also show that observed discharges are notably higher than model predictions possibility due to unmodeled late snowmelt effects. From August, the aggregated discharge values are more consistent across observed and model data, reflecting improved model performance. The relative skill of the model improves post-July as it aligns more closely with observed data, better capturing the persistence of low flows. Enhancing model calibration to account for regional hydrological events like late snowmelt or bias correcting precipitation data could improve forecast performance. It is worth mentioning that these types of discrepancies are expected when using global or regional models at local scales due to varying regional hydrological processes and model resolution limitations. 102 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 78 Visual inspection of EFAS forecasts from 2018 drought event for months of interest in terms of monthly total of discharges. Forecast initiation time is in the title of each subplot. When discussing the forecasts with users, the visual “eyeball” verification proved helpful for gathering practical feedback on how well the forecasts matched their expectations and needs (Figure 79). Users generally found the forecast discharge patterns to be coherent with their experience, especially during the summer months when low flows are most critical. They appreciated the clarity of the discharge graphs, noting that the daily and monthly representations improved the overall understandability of the forecast products. From NEA’s perspective, the discharge graphs seemed more useful across sectors, except for hydropower operators – who are mostly interested in aggregated volumes over time. Figure 79 Users perception on eyeball verification for the 2018 drought event in Georgia. 103 D3.4 User-driven Evaluation of the I-CISK Climate Services The graphs in Figure 80 illustrate the performance of the seasonal forecasts using Mean Absolute Error (MAE) and Continuous Ranked Probability Score (CRPS) across different lead times and discharge thresholds. Both metrics show a general decrease in error with longer lead times, although a slight peak at 2–3 months may reflect increased flow variability during that period. When normalised by the average minimum monthly discharge, the relative MAE is 27% for all data, 24% for discharges ≤12 m³/s, 22% for ≤10 m³/s, and 19% for ≤9 m³/s, suggesting higher forecast accuracy at low flows. While MAE provides a direct measure of forecast error magnitude, CRPS assesses the match between the forecast distribution and the observed outcomes. From a user perspective (Figure 80), the Mean Absolute Error (MAE) values were considered to be within an acceptable range—especially for low-flow forecasts at lower thresholds. In terms of credibility, users found the forecasts to be skillful up to a 3-month lead time, based on forecast errors. The Continuous Ranked Probability Score (CRPS), which measures the accuracy of the full forecast distribution, was more relevant from a forecaster’s perspective - users found this metric difficult to understand, which made it challenging to gather meaningful feedback. This points out the importance of using evaluation metrics that are both technically robust but accessible for users to interpret. Figure 80 Users’ perception on forecast errors - MAE and CRPS for different lead times and thresholds (EFAS). The ROC curves in Figure 81 illustrate that forecast quality decreases with increasing lead times across the different thresholds, as it approaches the diagonal (no skill). At a 1-month lead time, the forecasts are highly accurate, with ROC curves close to the top-left corner, indicating strong true positive rates and minimal false positives. As the lead time increases to 2 and 3 months, the forecast accuracy begins to decline with greater variability and spread in the ROC curves. By the 4th month, there is a noticeable drop in performance across all thresholds, with the ROC curves deviating further from the ideal position (top left corner). This trend continues through the 5th and 6th months, where the forecast accuracy significantly decreases, especially for the 12 m³/s threshold, which shows the highest false positive rates. It is important to note that these ROC curves are based on a very small number of observed events (e.g., 5 events), which makes the results highly sensitive. With such a limited sample, even one or two false positives or missed detections can drastically change the shape of the ROC curve. This can result in either very high apparent skill (if most rare events are correctly predicted) or very low skill (if even a few are misclassified). 104 D3.4 User-driven Evaluation of the I-CISK Climate Services Figure 81 ROC curves for different lead times and thresholds forecasts (EFAS). The contingency tables for 1-month, 3-month, and 6-month lead times with a 40% probability of nonexceedance are presented in Table 18, Table 19 and Table 20 respectively. The 40% threshold was selected based on visual inspection, as it appeared to offer a better balance between the probability of detection and false alarms — although this was difficult to assess due to the small number of observed events, especially at shorter lead times, where most points are aggregated in the top-left of the ROC plots. At a 1-month lead time, the forecast shows high accuracy in predicting non-events, with no false positives but some missed positive events, indicating low uncertainty. By the 3-month lead time, forecast uncertainty increases, evidenced by a balanced number of false positives and false negatives, and a decrease in true positives, reflecting reduced reliability in predicting positive events. At the 6-month lead time, while the detection of positive events improves, the forecast shows significant uncertainty with a substantial increase in false positives and a notable decrease in true negatives. Table 18. Contingency table for 1-month lead time forecasts (EFAS), 40% non-exceedance probability and 12 m³/s threshold. Observed/Predicted Positive (Predicted) Negative (Predicted) Row Total Positive (Observed) TP:2 FN: 3 5 Negative (Observed) FP: 0 TN: 74 76 Column Total 2 79 81 Table 19. Contingency table for 3-month lead time forecasts (EFAS), 40% non-exceedance probability and 12 m³/s threshold. Observed/Predicted Positive (Predicted) Negative (Predicted) Row Total Positive (Observed) TP:1 FN: 4 5 Negative (Observed) FP: 5 TN: 71 76 Column Total 3 75 81 105 D3.4 User-driven Evaluation of the I-CISK Climate Services Table 20. Contingency table for 6-month lead time forecasts (EFAS), 40% non-exceedance probability and 12 m³/s threshold. Observed/Predicted Positive (Predicted) Negative (Predicted) Row Total Positive (Observed) TP:4 FN: 1 5 Negative (Observed) FP: 46 TN: 30 76 Column Total 50 31 81 6.5 Discussions & Conclusions: Georgia Living Lab In Georgia, the co-evaluation process revealed that seasonal forecasts are seen as generally credible and useful by National Environmental Agency (NEA). Visualisation of individual forecasts suggested that the model had challenges in predicting hydrological components such as snowmelt as well as fast responding convective events in summer. This could be potentially addressed through bias correction of meteorological input (SEAS5) or streamflow and could be explored in further research. When analysing relative skill, results suggested that the model had overall moderate ability predicting low flow events for lead-times up to 3months, after which the skill decreased with high false positive rates compared to true positive rates. For short to medium term planning (1–3 months), a 40% non-exceedance threshold for low flows (e.g., 12 m³/s) is likely to be useful and actionable. For longer lead times, using a more conservative discharge threshold (e.g., 9 m³/s) or increasing the forecast probability threshold (e.g., to 50–60% non-exceedance) may help reduce false alarms. Another option could be complementing with expert judgement or additional sources of information to support decision-making under higher uncertainty. During the presentation of the reforecasts to the users, they appreciated the clear structure of the discharge forecast graphs and proffered visual tools like “eyeball” and event-based verification over more technical metrics. While MAE was perceived as intuitive and aligned with their understanding of forecast accuracy, more complex metrics such as CRPS and ROC curves were found difficult to interpret, showing the need to simplify communication or provide additional guidance. Forecast skill was considered as useful up to around three months, especially for low-flow thresholds, which are most relevant during the summer decisionmaking period. These findings show that for forecasts to hold real value, they must be not only credible, but also salient (i.e. relevant to user needs) and legitimate (i.e. trusted and developed through inclusive processes). Even highly accurate forecasts may be disregarded if they do not align with users’ specific decision contexts, priorities, and timing. In Georgia, the co-evaluation process—especially the participatory nature of the session— contributed to trust-building and highlighted that forecast value is shaped as much by user relevance as by technical skill. Finally, the co-evaluation also revealed that regular engagement with forecasts and iterative dialogue with users could improve their confidence and therefore support more effective drought preparedness. Still, participants mentioned the need for more time, broader inclusion of users, and space for participatory sessions to further strengthen the service. 106 D3.4 User-driven Evaluation of the I-CISK Climate Services 7 Discussion This study has applied a general framework to evaluate six climate services from a user-driven perspective. Some common challenges arose when applying this framework in the living labs. The most significant challenge was to define the thresholds which were associated with users’ decision-making. In some living labs, such as Spain, users were able to define specific threshold values, for example the accumulated rainfall associated with normal conditions. However, some of these thresholds were not consistent with the conditions they purported to represent when observed values from in-situ time series data were analysed. In Spain for example, the user defined by stakeholders threshold for ideal rainfall accumulation was rarely reached in the in-situ observed time series. This highlights the importance of evaluating any user-defined thresholds prior to beginning the climate service evaluation. Another challenge faced when using user defined thresholds was the issue of bias in the evaluated climate service. A positively biased climate service, for example, would likely under-estimate the number of drought or low streamflow events if the threshold was derived by users using in-situ observations. One solution would be to bias correct the climate service using in-situ observations. However, bias corrected climate services were not available in all living labs. In these situations, this study found it necessary to recompute the thresholds directly from the climate service’s re-analysis or re-forecast time series. The challenge was to compute a threshold which would be meaningful to the user. This could be achieved through the following steps: 1) computing different thresholds for a range of percentiles, 2) applying each threshold to the climate service’s re-analysis time series, if one was available, to define an event time series, 3) comparing each event time series against the number of observed events provided by the user and selecting the threshold which produces the best agreement. However, this relies on user provided time series of observed events, which was not always available. Defining observed events was another challenge as these datasets were not available in every evaluated living lab. Such a dataset, with a sufficient number of observed events, is essential in order to perform a robust evaluation. To solve this issue, a proxy observation time series could be defined from the climate service’s re-analysis dataset, if one was available. Where possible, these derived proxy observations should be compared against any observations which have been provided by the user, this is to determine their trustworthiness. If this is not possible, it should be communicated to users that the evaluation was only possible against re-analysis based data. When the evaluation results were presented to end users, the user driven metrics, such as hit rate and threat score, were received more positively by users than results from conventional statistics scores such as the Mean Absolute Error (MAE). Users also expressed a desire to see the breakdown of the hits, misses and false alarms in addition to the user driven metric. This provided greater transparency about the calculation of score and gave a better idea of what they could expect from the climate service, for example how frequently it may produce false alarms. Users also liked being presented with examples of the climate service performance during specific events of interest. This helped demonstrate to users the potential benefit of the climate service to users, as they can compare its guidance against their own experience when the climate service may not have been available. 7.1 A revised user-centred evaluation framework Based on the findings from the user driven evaluation in this study and the discussions above, it is necessary to revise the general evaluation framework which was outlined in the introduction (Figure 82). The revisions come after the first step of receiving users’ inputs. The first revision is creating a database of impactful 107 D3.4 User-driven Evaluation of the I-CISK Climate Services events, which could be derived either from in-situ gauge observations or climate service re-analysis data. The database should contain a sufficient number of events over several years to allow for a robust evaluation. Secondly, the thresholds provided by the users should be evaluated against the database of impactful events to determine if the thresholds themselves are realistic, and whether they may be affected by biases in the data that is used in the climate service. If either situation arises, the thresholds should be computed directly from the climate service’s re-analysis dataset. Figure 82 Revised general framework for performing user-driven evaluation of climate services. This study did not address the final two steps of the proposed evaluation framework, which focussed on incorporating the findings directly into the operational aspects of the climate service. In some of the analysed living labs, such as Lesotho, results were transformed into guidance about how the users could use the climate services to make decisions. Unfortunately, we did not receive feedback about whether users have now adopted this guidance operationally. The incorporation of user driven evaluation information into climate services could be achieved by adapting the visualisation of the climate service. Currently the climate service forecasts are visualised as time series of a forecasted variable, such as temperature. The visualisation could be modified to also display the threshold, identified during the user-driven evaluation, at which a user should take action. This would allow a user to quickly identify whether they should take action based on a climate service forecast. This idea is also discussed in the I-CISK deliverable 3.5 about climate service visualisation (Van Andel et al., 2025) and could be a topic for future development. 108 D3.4 User-driven Evaluation of the I-CISK Climate Services (2) roads reported in neighbouring South African provinces. border. Couldn’t find any reports for Lesotho. May 2007 21/05/ 2007 26/05/ 2007 6 Severe Danger to life 21 deaths were reported in South Africa; the locations were not clear but the same report detailed snow in provinces Neighbouring Lesotho. https://www.met.reading.ac.uk/ ~brugge/world2007.html Unclear the exact impacts in Lesotho (as opposed to SA) June 2007 27/06/ 2007 28/06/ 2007 2 (+) Severe Danger to Life, Transport Rescue operation required from vehicles and trucks on the Lesotho SA border. https://www.iol.co.za/news/southafrica/snow-wreaks-havoc-acrosssa-359626 https://mg.co.za/article/2007-0627-cold-snap-snow-claim-life-injohannesburg/ https://mg.co.za/article/2007-0627-snow-covers-rooftops-ingauteng/ https://2001-2009.state.gov/r/pa/ ei/pix/b/af/les/ 87473.htm#:~:text=An %20unexpected%20delight%2C %20snow%20covered,nearby %20mountains%2C%20per%20the %20Embassy. End date unclear – from report on 28th it looks like the event could have gone on for quite a bit longer. Many more impacts reported across SA, widespread transport issues included delayed flights at Johannesburg OR Tambo airport. Septem 19/09/ 22/09/ 4 Minor Infrastructure Electricity went out https://ttl-lesotho.org/ Dates are a combination of 115 D3.4 User-driven Evaluation of the I-CISK Climate Services ber 2008 2008 2008 during snowstorm in Mokhotlong 2008/09/22/snow-2/ https://natalia.org.za/Files/43/ Natalia%2043-Article%20Alcock %20pp%2073%20to%2083.pdf the impact report and a report of snow falling in the area (both provided in the references). June 2011 08/06/ 2011 10/06/ 2011 3 Significan t Transport Roads closed for several days; Tourists trapped in Lesotho snow Grab and Linde 2013, citing ‘Personal communication with Afriski Resort’ EAP, potentially from the same original source End date estimated - 'several days' duration given in report taken as 3 July 2011 25/07/ 2011 26/07/ 2011 2 (+) Severe Danger to life, Livelihood, Transport Air rescue operation required; Loss of communications; Damage to buildings; Major disruption to roads (and shipping in SA). New internet trawl: https://www.met.reading.ac.uk/~br ugge/world2011.html https://mg.co.za/article/2011-0726-snowfall-brings-sa-to-astandstill/ https://awanderingjo.blogspot.com /2011/08/drakensburg-and-malutimountains-are.html https://www.pih.org/article/asnowstorm-in-july" Duration may have been longer and the reports don't document the end of the impacts well. July 2012 14/07/ 2012 15/07/ 2012 2 Severe Danger to life, Transport Rescue from vehicles required; [Two deaths in South Africa, but none reported in Lesotho] New internet trawl: "https://lesothoblanketwrap.com/2012/news/chao s-in-the-snow-as-winter-storms-hitthe-mountain-kingdom-of-lesotho/ https://www.aljazeera.com/news/2 Nil 116 D3.4 User-driven Evaluation of the I-CISK Climate Services 012/7/17/stormy-weather-blastssouthern-africa" "https://www.4x4community.co.za/ forum/showthread.php/119626PLEASE-HELP-stuck-in-snow-inLesotho https://snowreport.co.za/historyof-snow-in-south-africa-after-2010/ " August 2012 07/08/ 2012 10/08/ 2012 13 Significan t Livelihood, Transport Roads closed; Mine unable to operate Grab and Linde 2013, citing personal communications with Afriski Resort, but no date specified in August 2012 provided New internet trawl: https://snowreport.co.za/historyof-snow-in-south-africa-after-2010/ https://www.washingtonpost.com/ blogs/capital-weather-gang/post/sn ow-blankets-parts-of-south-africaa-rare-dusting-in-johannesburg/ 2012/08/07/8ef90e52-e0ad-11e1a19c-fcfa365396c8_blog.html https://www.ctvnews.ca/world/rar e-snowfall-stuns-much-of-southafrica-1.906032 https://www.4x4community.co.za/f orum/showthread.php/121949Lesotho-Snow-7-Aug-2012 https://rustysodyssey.blogspot.com /2012/08/august-7-2012-snow-inlesotho.html https://www.met.reading.ac.uk/~br End date is an estimate; two records close together treated as the same event, although one does not mention impacts. The EAP entry for 07-11 August 2012 appears to have been incorrectly dated. There is a news report that uses almost exactly the same wording, which dates the incidents referred to on the Butha Buthe pass as having occurred on the 15 July 2012 (https://lesothoblanketwrap.com/2012/new s/chaos-in-the-snow-aswinter-storms-hit-themountain-kingdom-oflesotho/ ). 117 D3.4 User-driven Evaluation of the I-CISK Climate Services ugge/world2012.html https://www.snow-forecast.com/re sorts/AfriSki/photos/19914 July 2016 23/07/ 2016 27/07/ 2016 5 Severe Danger to life, Livelihood, Transport Eight human lives lost; Livestock deaths; Air rescue operation required EAP, citing the Weather Channel, 2016 (however, see note below on dates for this event). New internet trawl: https://snowreport.co.za/historyof-snow-in-south-africa-after-2010/ https://www.earthobservatory.nas a.gov/images/88509/lesotho-seesits-heaviest-snowfall-in-twodecades https://www.sott.net/article/32484 1-Several-dead-after-heaviestsnow-in-two-decades-hits-LesothoAfrica https://weather.com/news/news/le sotho-south-africa-snow-twodecades Duration of event from a combination of consecutive snow reports and an article that states 'heaviest snow in two decades blanketed Lesotho in late July 2016', though main impacts are all reported on the 27th. The dates provided for this event in the EAP (03/07/2016-04/07/2016) are incorrect, which can be seen if by cross checking the reference provided in the EAP table (https://weather.com/news/ news/lesotho-south-africasnow-two-decades). Also of note, the deaths reported are for a South African district that borders Lesotho, so it’s not clear exactly where these happened. May 2017 12/05/ 2017 12/05/ 2017 1 Minor Transport Roads impassable in Maseru New internet trawl: https://www.jacarandafm.com/sho ws/the-workzone-with-alex-jay/pict ures-first-snow-fall-south-africa/ Nil May 2018 31/05/ 2018 31/05/ 2018 1 Minor Transport Roads closed EAP, citing https://www.tourismupdate.co.za/ EAP end date is in August so it wasn't used. 118 D3.4 User-driven Evaluation of the I-CISK Climate Services article/snow-lesotho-causes-roadclosure Octobe r 2018 03/10/ 2018 04/10/ 2018 2 Significan t Livelihood, Transport Thousands of livestock dead; At least one road closed EAP, citing the Lesotho Meteorological Service New internet trawl: https://www.citizen.co.za/news/so uth-africa/in-tweets-snow-falls-onpicturesque-drakensberg/ The impacts in the EAP are listed generically as ‘October’ but are assigned to this event as it is the only other October event found that has impacts (minor, transport) Note that the EAP identifies this as out of season. April 2020 25/04/ 2020 29/04/ 2020 5 Minor Transport Road restrictions EAP citing Independent online Start end dates are the beginning of the EAP record and the date the article it cites was published. June 2020 10/06/ 2020 16/06/ 2020 7 Minor Transport Road infrastructure blocked for a week EAP, citing the Lesotho Meteorological Service Nil August 2021 (1) 14/08/ 2021 15/08/ 2021 2 Significan t Infrastructure, Livelihood, Transport Communities cut off; Power outages; Tourism industry disrupted; Road closures and traffic disruption EAP, citing the Lesotho Meteorological Service and the following links: https://www.news24.com/news24/ southafrica/news/pics-lesotholodge-blanketed-in-snow-as-icyweather-sweeps-sa-20210814 https://newscentral.africa/parts-oflesotho-blanketed-in-snow/ New internet trawl: https://www.sapeople.com/fabsouth-african-stuff/watch-it-snowdown-in-africa/ Nil August 26/08/ 27/08/ 2 Significan Livelihood Livestock deaths EAP, citing Nil 119 D3.4 User-driven Evaluation of the I-CISK Climate Services 2021 (2) 2021 2021 t https://mkweather.com/disruptivesnowfall-in-south-africa-regionallyup-to-50-cm/#google_vignette May 2022 21/05/ 2022 21/05/ 2022 1 Minor Transport Disruption on roads New internet trawl: https://www.citizen.co.za/news/so uth-africa/parts-of-south-africaexperience-first-winter-snowfall/ EAP also includes the event, but does not record any impacts. 120 D3.4 User-driven Evaluation of the I-CISK Climate Services Appendix 2 Glossary Acronym Definition API Application Programming Interface C3S Copernicus Climate Change Service CDS Climate Data Store CEMS Copernicus Emergency Management Services CMIP World Climate Research Programme’s Coupled Model Intercomparison Project CORDEX Coordinated Regional Climate Downscaling Experiment CS Climate Services CSIS Climate Services Information Systems DRR Disaster Risk Reduction GEO Group on Earth Observations GEOSS Global Earth Observation System of Systems GUI Graphical User Interface IPCC Intergovernmental Panel on Climate Change LL Climate Services Living Labs NHMS National Hydro-meteorological Service MOOC Massive Open Online Course OGC Open Geospatial Consortium S2S Sub-seasonal to Seasonal TRL Technology Readiness Level UNCCD United Nations Convention to Combat Desertification UNDRR United Nations Office for Disaster Risk Reduction UNFCCC United Nations Framework Convention on Climate Change WCRP World Climate Research Programme WFD Water Framework Directive WMO World Meteorological Organization 1 D3.4 User-driven Evaluation of the I-CISK Climate Services Colophon: This report has been prepared by the H2020 Research Project “Innovating Climate services through Integrating Scientific and local Knowledge (I-CISK)”. This research project is a part of the European Union’s Horizon 2020 Framework Programme call, “Building a low-carbon, climate resilient future: Research and innovation in support of the European Green Deal (H2020-LC-GD-2020)”, and has been developed in response to the call topic “Developing end-user products and services for all stakeholders and citizens supporting climate adaptation and mitigation (LC-GD-9-2-2020)”. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293. This four-year project started November 1st 2021 and is coordinated by IHE Delft Institute for Water Education. For additional information, please contact: Micha Werner ([email protected]) or visit the project website at www.icisk.eu °°°° This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293