Bird migration model 2.0: implemented changes and comparison with Model 1.1

Bird migration model 2.0: implemented changes and comparison with Model 1.1 Maja Bradarić, Bart Kranstauber and Judy Shamoun-Baranes University of Amsterdam, Institute for Biodiversity and Ecosystem Dynamics, PO Box 94240, 1090 GE Amsterdam Project manager: prof. dr. Judy Shamoun-Baranes Date of publication: June 2025 The report should be cited as: Bradarić M, Kranstauber B & Shamoun-Baranes J. 2025. Bird migration model 2.0: implemented changes and comparison with model 1.1. University of Amsterdam. This work was supported by the Dutch Ministry of Infrastructure and Water Management - Rijkswaterstaat (project number 31184728) through "Start-stop project" and "Wind at sea" (WOZEP) programmes. 2 Contents Summary ................................................................................................................................ 3 Background ............................................................................................................................ 4 Implemented changes .............................................................................................................. 5 ERA5 data ........................................................................................................................ 5 Predictor variables ............................................................................................................... 6 Phenology ....................................................................................................................... 7 Accumulation at departure ................................................................................................ 9 Comparison Model 1.1. and Model 2.0 .................................................................................... 11 Main model evaluation ....................................................................................................... 12 Conclusions .......................................................................................................................... 14 1. Elements explored but ultimately not included in the model ............................................... 16 Using data from highly correlated radar locations to fill in the gaps in the data ...................... 16 Extension of the prediction period .................................................................................... 16 Smaller number of predictor variables .............................................................................. 16 Include weather conditions at departure in Norway as predictors ........................................ 16 Weights ......................................................................................................................... 16 2. Data vs predictions time series ......................................................................................... 17 Spring ........................................................................................................................... 17 Autumn ......................................................................................................................... 19 3. ROC curves .................................................................................................................... 21 Spring ........................................................................................................................... 21 Autumn ......................................................................................................................... 22 4. Cumulative migration plots .............................................................................................. 23 Spring ........................................................................................................................... 23 Autumn ......................................................................................................................... 24 3 Summary In this report, we provide an overview of the changes implemented in the latest bird migration prediction model, Model 2.0, which are the result of additional research conducted at UvA (University of Amsterdam) and input from the teams manually predicting migration for the start/stop procedure, conducting radar data analysis (Waardenburg Ecology) and operationalizing the model predictions (Technolution). Changes implemented in the Model 2.0 development occurred on multiple levels and include datalevel changes, selection of predictor variables, and structural model changes. The main changes include: 1. Longer time series (Model 2.0 trained with spring and autumn data from Luchterduinen radar location between 2019 and 2024) 2. Data augmentation (SMOTE) was performed for balancing the data 3. Adapted bird accumulation variables to introduce estimated bird departure variables 4. Adapted phenology variables 5. Excluded variables that represented nightly differences in bird accumulations due to wind assistance and precipitation 6. Adjusted model hyperparameters Model 2.0 performs considerably better than Model 1.1 in terms of general accuracy, variance explained, as well as precision and recall. The level of improvement varies from year to year, but is visible in all test years. The general increase in precision and recall resulted in fewer false negatives and a higher number of true positives. A higher number of false positives is observed as a direct consequence of applying the SMOTE technique. We demonstrate that across testing years, the percentage of migrants saved within 30 hours of the highest intensity migration with Model 2.0 is several times higher in both seasons than when predicting with Model 1.1. Nevertheless, there is still room for future improvement, which will likely occur with the collection of more data and the development of improved technologies for tracking birds at sea using radars to decrease the periods during which the data is affected by various forms of clutter. Alternatively, modelling approaches that take a migration system-wide approach might be beneficial, but are currently operationally challenging. 4 Background As part of the start-stop project, coordinated by Rijkswaterstaat, the bird migration prediction model is being used to inform curtailments of offshore wind turbines in the Dutch North Sea during intense bird migration events. The first version of this model (Model 1.0) was delivered in 2022 and replaced with the second version (Model 1.1) in 2023, after additional radar data were collected for model training. The model's performance is evaluated on a half-yearly basis by an independent team of ecologists (Waardenburg Ecology), and its operation is closely monitored by data scientists (Technolution), as well as the team of ornithologists responsible for independent migration predictions. All these parties evaluate model performance under different migration events and give recommendations for potential model improvements. Concurrently, more radar data have been collected offshore for model training and evaluation. Data availability, especially the availability of moments of intense migration (which generally occur on a few nights during a migration season), has been identified as one of the primary challenges in achieving accurate predictions of intense bird migration (Bradarić et al, 2024; Middelveld et al, 2023; Kuijper et al, 2025). Therefore, longer data timeseries are generally expected to improve model performance. Since the development of Model 1.0, a team of UvA scientists conducting interdisciplinary research that combines ecology, meteorology, and data science has further explored migration patterns across the North Sea. This was made possible by the deployment of additional offshore radars at various locations in the North Sea. Besides providing the first overview of broader spatiotemporal bird migration patterns across the North Sea, the insights collected during this research have contributed to identifying the adaptations needed for further model development (Bradarić & Shamoun-Baranes, 2025). One of the aspects explored during this research was whether a separate model for the Zeeland region is necessary. High correlations across radars in Luchterduinen and the Zeeland region indicated that the patterns are quite comparable (Bradarić & Shamoun-Baranes, 2025). Patterns at the Gemini location showed that a separate model might be beneficial for the region around the Gemini wind park, as the radar at this location generally has low correlation with radars in the southern North Sea (Bradarić & Shamoun-Baranes, 2025). However, the model development for the Gemini region was out of the scope of the current project. Therefore, using novel research findings, input from various expert teams, and additional data, the team at UvA has developed a new bird migration prediction model (Model 2.0) based on data collected at the Luchterduinen location, which provides a good representation of migration patterns across the southern North Sea. The changes and updates applied to Model 2.0 are presented in this report. The primary goal of developing the new model version was to increase model accuracy, particularly when predicting intense migration. 5 Implemented changes Changes implemented in Model 2.0 occurred on multiple levels and include data-level changes, changes in predictor variables, and structural model changes. Below, we provide a more detailed explanation of these changes. Data Timeseries length For Model 2.0 training, an extra year of data (spring and autumn 2024) was used. Longer data time series improve model accuracy primarily through contributing more moments of intense bird migration from which the model can learn. Model 2.0 has been trained using radar data from the Luchterduinen radar location (52.25 N, 4.10 E), collected during the period of peak migration in spring (15 February – 30 April) and autumn (1 October – 30 November) of 2019 – 2024, a year longer than Model 1.1. The shortened migration season aims to reduce noise in the data and focus on the period of intense migration; all peaks occurred during the periods included in the model training (Bradarić et al., 2024; Bradarić & Shamoun-Baranes, 2025). Imbalanced data and Synthetic Minority Oversampling Technique (SMOTE) Due to the nature of bird migration, which typically has periods of low migration and a few pulses of intense migration, the training data used is highly imbalanced, containing many more hours of lowintensity migration than those with high-intensity migration. Shortening the migration period used for model training was one way to decrease the imbalance in the data (Bradarić et al, 2024). However, even after doing that, the training data still has only a few moments of peak migration. As the aim is to correctly predict peak moments while limiting false positive predictions, methods for data balancing needed to be applied. In Models 1.0 and 1.1, this was addressed through stratified sampling (see Bradarić et al., 2024). In Model 2.0, as suggested by Technolution (Kuijper et al, 2025), the synthetic minority oversampling technique (SMOTE) adjusted for regression was employed. Generally, SMOTE is a type of data augmentation where the minority subset of the data is oversampled. Here, we used the SmoteRegress function from the UBL R package (Branco et al, 2016). The algorithm generated synthetic data points for underrepresented cases (high-intensity migration) by interpolating between existing instances, using a Manhattan distance metric to identify nearest neighbours. The threshold for defining a rare (minority) case was set to 0.95, meaning that any observation above the seasonal 95th percentile of MTR data was considered rare and oversampled. We used an oversampling rate of 20, which means that for every data point of the rare observations (high-intensity migration), 20 synthetic ones were created. In contrast, the majority group (lowintensity migration) stayed unchanged. This was implemented before model training for the spring and autumn migration seasons separately.. ERA5 data ERA5 reanalysis data were re-downloaded, and weather variables for all years recalculated using the same methodology as described in Bradarić et al 2024. This was necessary due to potential changes in variable calculations due to updates of models that are generating ERA5 data. The data processing code was also updated to accommodate changes in data storage applied by ECMWF. 6 Predictor variables After testing different combinations and numbers of predictor variables, the best model performance was obtained with the subset of variables used in Model 1.1. All variables that represented nightly differences in accumulation due to wind assistance and precipitation were replaced with estimated departures due to wind assistance and precipitation at the respective departure locations. These estimated departure variables represent proportions of birds that accumulated in the previous night and the theoretical numbers of birds arriving on the respective night (based on seasonal phenology in Figure 1). The theoretical accumulation and migration depend on the wind assistance and precipitation. For more details, see the accumulation section. See Table 1 for a complete list of predictor variables included in the model training process. Table 1. Overview of variables used for model training in spring and autumn. Variables with the asterisk (*) next to them represent those that were amongst the top 10 variables by variable importance (those most contributing to the model) in all combinations of training years. Spring Autumn Hourly phenology* Hourly phenology* Yearly phenology* Yearly phenology WA towards NE radar location* WA towards SW at radar location* WA towards E radar location* WA towards W at radar location* Total precipitation at the radar location Total precipitation at the radar location Wind assistance in the UK Mean sea level pressure at the radar location* Total precipitation in the UK Total precipitation in N Netherlands Nightly difference in mean sea level pressure in the UK Estimated departure due to precipitation in N Netherlands* Estimated departure due to WA in the UK* Temperature at N Netherlands Estimated departure due to precipitation in the UK Estimated departure due to WA in N Netherlands* Estimated departure due to WA and precipitation in the UK* Estimated departure due to WA and precipitation in N Netherlands* Accumulation due to precipitation in the UK Accumulation due to WA in N Netherlands Wind assistance in France Accumulation due to precipitation in N Netherlands Total precipitation in France Total precipitation in NW Germany Mean sea level pressure in France* Mean sea level pressure in NW Germany Temperature in France Nightly difference in mean sea level pressure in NW Germany Estimated departure due to WA in France Estimated departure due to WA in NW Germany Estimated departure due to precipitation in France Estimated departure due to precipitation in NW Germany Estimated departure due to WA and precipitation in France Estimated departure due to WA and precipitation in NW Germany Accumulation due to WA in NW Germany Accumulation due to precipitation in NW Germany Wind assistance in Denmark Temperature in Denmark Estimated departure due to WA in Denmark Estimated departure due to precipitation in Denmark Estimated departure due to WA and precipitation in Denmark Accumulation due to WA in Denmark* Accumulation due to precipitation in Denmark 7 Phenology Seasonal and nightly phenology variables were updated to reflect a 6-year seasonal bird migration pattern. Generalised additive models were fit to data using day of year and hour after sunset as a predictor variable, and MTR as a response variable, as described in Bradarić & Shamoun-Baranes, 2025. Updated seasonal (Figure 3) and hourly (Figure 4) phenology curves are presented below. These curves are also used to estimate the numbers of birds arriving at departure locations in accumulation functions as described above. Figure 1. Seasonal phenology curves in spring (upper panel) and autumn (lower panel). The black line represents the modelled relationship between MTR (birds/km/hr) and day of year. The grey band denotes 95% confidence intervals, which indicate variation among nights. 8 Figure 2. Hourly phenology curves in spring (upper panel) and autumn (lower panel). The black line represents the modelled relationship between MTR (birds/km/hr) and hours after sunset. The grey band denotes 95% confidence intervals, which indicate variation among nights. 9 Accumulation at departure When weather conditions are not favourable for migration for a few days, birds that would otherwise depart may accumulate in large numbers and depart simultaneously once weather conditions improve. This has been observed along coastlines for birds that need to embark on a journey across the sea (Buler & Moore, 2011). To simulate accumulation behaviour, Models 1.0 and 1.1 simulate a fraction of accumulated birds due to wind assistance and precipitation at various departure locations. A binary threshold was used to estimate when 2/3 of the birds that would otherwise depart will accumulate following Erni et al, 2002. In Model 2.0, we updated accumulation functions to estimate the number of birds accumulating and departing each night, depending on weather conditions and seasonal phenology (Figure 1), which serves as a long-term average and a proxy for the number of birds passing through. This adaptation incorporates one of the suggestions from the bird migration prediction team to include an estimate of birds flying over the radar location (note this is not based on the actual observations of birds flying by), thereby avoiding predictions of intense migration when most birds have already departed. a) b) Figure 3. a) Gaussian (bell-shaped) curve describing departures due to wind assistance in spring and b) sigmoid function describing departures due to wind assistance in autumn. 16 Appendix 1. Elements explored but ultimately not included in the model Stricter data filtering It has been suggested by Waardenburg Ecology to include stricter filtering thresholds, which would exclude more data. As this step would exclude even more hours from an already noisy dataset, we decided against it now. Using data from highly correlated radar locations to fill in the gaps in the data This has been tested between Borssele and Luchterduinen, as well as between HKZA and Luchterduinen radars. The test showed that they usually lacked data at the same moments, and it was decided no to implement this in the modelling procedure. Extension of the prediction period As the model is being run throughout the full bird migration period (February 15 to May 31 in spring; August 15 to November 30 in autumn), it was discussed to also train it with the data from the full migration period instead of focusing on the period when most of the intense migration occurs (February 15 to April 30 in spring; October 1 to November 30 in autumn). In discussion with RWS, it was decided to limit the model training data to the periods when we expect the most intense bird migration, in order to reduce the amount of noise in the data and keep the training data more balanced. Smaller number of predictor variables Technolution has suggested that decreasing the number of predictor variables might improve model performance, as the number of variables used in Model 1.1 might have been too high for the amount of data used for training (Kuijper et al, 2025). Therefore, the model was set up with varying combinations and numbers of variables, and the model performance was tested. Decreasing the number of variables generally decreased the model performance, and the variables included in Model 2.0 (Table 1) were the ones which resulted in the best overall model performance. Additionally, the number of observations in the training data used in Model 2.0 was increased by the SMOTE technique, considerably increasing the size of the dataset, and the number of variables is more in line with the dataset size. Include weather conditions at departure in Norway as predictors We downloaded and calculated all departure area predictor variables for the Norway location and tested the model performance after their inclusion in the predictor variables. Different combinations of models, which contained predictor variables from departure areas in Norway, showed inferior model performance to the setup we currently use for Model 2.0. Weights Adding weights to observations during the modelling procedure (e.g. favouring observations of intense-migration hours) can help increase model performance when dealing with imbalanced datasets. However, when the SMOTE technique is already applied, adding weights can be counterproductive and can lead to model overfitting. We tested adding weights to predictor variables, but this resulted in inferior model performance to the setup we currently use in Model 2.0. 17 2. Data vs predictions time series Seasonal time series of hourly values predicted by Model 1.1 (old) and Model 2.0 (new) compared against observed values (black line) in different testing years. Spring 18 19 Autumn 20 21 3. ROC curves The receiver operating curves (ROC) for the spring and autumn models were created with a threshold of 150 birds/km/h in spring and 173 birds/km/h in autumn (current thresholds used in operation). Red lines represent the performance of the new model (Model 2.0), and the green lines represent the old model (Model 1.1). These plots demonstrate the model's effectiveness in identifying hours with high migration intensity while minimising the fraction of false positive predictions. Spring 22 Autumn 23 4. Cumulative migration plots The figures below show the cumulative percentage of total nocturnal migration during hours in spring and autumn, for which the radar data were available. The percentage of seasonal migration in each available migration hour was calculated and then summed to get a cumulative percentage. The black line represents the cumulative percentages when the measured migration hours were ranked based on the measured MTR in descending order (hours with the highest measured MTR in the left corner of the figure). The green and red lines represent the cumulative percentages of measured MTR when the migration hours are ranked in descending order based on the MTRs predicted by Model 1.1 (green) and Model 2.0 (red). Note that the values represented by green and red lines are the measured hourly percentages of the total seasonal migration. Numbers in plots represent the percentages of birds saved by each of the models at a 30-hour cutoff. Spring 24 Autumn