Life Outcome Prediction with Foundation Models Trained on Population Registry Data

Ana Macanovic Complexity Science Hub Vienna 19.11.2025. Life Outcome Prediction with Foundation Models Trained on Population Registry Data Today’s talk 1. Predicting life outcomes - Wrong modelling choices? - Too little data? 2. Foundation transformer models for life course modelling - Building sequences - Enrichments 3. Model evaluation 4. Interesting stuff 5. Where do we go from here? https://www.flaticon.com/ Icons by Peerapak Takpho and Becris Team Tanzir Pial Lucas Sage Flavio Hafner Enamul Hassan Tom Emery Arnout van de Rijt Steven Skiena Dakota Handzlik 1. Predicting life outcomes - Knowing what will happen to individuals in our societies can: - improve policies - improve understanding of differences - boost our theoretical understanding of the social life https://www.flaticon.com/ Icons by Freepik and Nualnoi Kinkaeo 1. Predicting life outcomes - Knowing what will happen to individuals in our societies can: - improve policies - improve understanding of differences - boost our theoretical understanding of the social life - What might we want to know? - Demographic indicators - Sociological indicators - Attitudes, behaviours - Health outcomes https://www.flaticon.com/ Icons by Freepik and Nualnoi Kinkaeo 1. Predicting life outcomes https://www.flaticon.com/ Icons by Freepik and Nualnoi Kinkaeo 10 years from now: 18.02 –19.71 20 years from now: 18.69 –20.23 1. Predicting life outcomes - How do we usually do this? - Specialized models for individual tasks: - Demographic predictions - Logistic regression - Machine learning (e.g., Random Forest) https://www.flaticon.com/ Icons by Peerapak Takpho and Becris fertility income divorce relevant data relevant data relevant data 1. Predicting life outcomes is hard! 1. Predicting life outcomes is hard! - Extrapolation from the last period best for fertility (Bohk-Ewald 2018) 1.a. Modelling lives: Life course theory - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd no job job 1 job 2 - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) 1. Life domains are entangled https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd single married married & child 1.a. Modelling lives: Life course theory no job job 1 job 2 - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) 1. Life domains are entangled 2. Life sequences exhibit path-dependence and lock-in 1.a. Modelling lives: Life course theory single married married & child no job job 1 job 2 single no job job 1 no job https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) 1. Life domains are entangled 2. Life sequences exhibit path-dependence and lock-in 3. Timing and duration matter https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd 1.a. Modelling lives: Life course theory no job job 1 job 2 no job job 1 job 2 no job job 1 job 2 - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) 1. Life domains are entangled 2. Life sequences exhibit path-dependence and lock-in 3. Timing and duration matter 4. Lives are “linked” https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd job 1 no job 1.a. Modelling lives: Life course theory single married married & child no job job 1 job 2 - Linking outcomes to the ordering and duration of a range of interrelated life events (Willekens 1999) 1. Life domains are entangled 2. Life sequences exhibit path-dependence and lock-in 3. Timing and duration matter 4. Lives are “linked” - Sequence analysis techniques too restrictive to account for these factors https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd healthy sick 1.a. Modelling lives: Life course theory job 1 no job single married married & child no job job 1 job 2 1.b. More data: CBS Register data https://www.flaticon.com/ Icons by Freepik - Dutch Register Data - All residents of the Netherlands - Continuously updated - Some yearly datasets go back to early 2000s, most to 2010s - Our dataset -~23 million people -~7.5 billion records 1.b. More data: CBS Register data https://www.flaticon.com/ Icons by Freepik - Dutch Register Data - All residents of the Netherlands - Continuously updated - Some yearly datasets go back to early 2000s, most to 2010s - Our dataset -~23 million people -~7.5 billion records background/ household education employment marriage/ divorce network income 2. Potential answer? sequences fertility income divorce https://www.flaticon.com/ Icons by Freepik background/ household education employment marriage/ divorce network income 2. Potential answer? sequences fertility income divorce https://www.flaticon.com/ Icons by Freepik background/ household education employment marriage/ divorce network income 2. Transformer models: Bonus 1: Parents and partners https://www.flaticon.com/ Icons by Anditii Creative and Becris - Incorporate background and current events of parents into sequences up to 18 years of age - Incorporate background and current events of partners throughout the duration of registered partnership/marriage 2. Transformer models: Bonus 2: Population-scale networks! https://www.flaticon.com/ Icons by Anditii Creative and Becris -Multiplex population-level opportunity network - family, household, neighbors, classmates, colleagues 2. Transformer models: Bonus 2: Population-scale networks! https://www.flaticon.com/ Icons by Anditii Creative and Becris -Multiplex population-level opportunity network - family, household, neighbors, classmates, colleagues -DeepWalk (Perozzi et al, 2014) with layer persistence and hubs - Temporal alignment across years 2. Transformer models: Bonus 2: Population-scale networks! https://www.flaticon.com/ Icons by Anditii Creative and Becris -Multiplex population-level opportunity network - family, household, neighbors, classmates, colleagues -DeepWalk (Perozzi et al, 2014) with layer persistence and hubs - Temporal alignment across years - Generalized Fibonacci grid method for higher dimensions to “cluster” embeddings 2. Transformer models: Bonus 2: Population-scale networks! https://www.flaticon.com/ Icons by Anditii Creative and Becris -Multiplex population-level opportunity network - family, household, neighbors, classmates, colleagues -DeepWalk (Perozzi et al, 2014) with layer persistence and hubs - Temporal alignment across years - Generalized Fibonacci grid method for higher dimensions to “cluster” embeddings - Including embedding cluster as a sequence token in a given year 2. Transformer models: Model pretraining https://www.flaticon.com/ Icons by Freepik -BERT architecture - Different sizes: 8 and 80 million parameters (tested up to 540) - Training objectives: Masked Language Modelling (MLM) and Sequence Order Prediction (SOP) -240-dimensional embeddings - Train on sequence data of 23 million people up to 2020 3. Model evaluation https://www.flaticon.com/ Icons by Freepik -Prediction tasks: - Demographic events: fertility, divorce (7 mln/200k people) - Socioeconomic status: income (200k people) - LISS survey: ethnic self-identification, motor vehicle ownership 3. Model evaluation https://www.flaticon.com/ Icons by Freepik -Two approaches: - Feed static embeddings into a fully connected NN with 1/2 layers - Fine-tune the model on the task (update the weights) -Baseline: linear regression with all variables used for pre-training 3. Model evaluation: Fertility -PreFer challenge –predicting whether a person will have a child in the next 3 years F-1 scores of our different models on the Predicting Fertility Challenge. https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd 3. Model evaluation: Fertility -PreFer challenge –predicting whether a person will have a child in the next 3 years F-1 scores of our different models on the Predicting Fertility Challenge. https://www.flaticon.com/ Icons by Vlad Szirka, Freepik, bsd 3. Model evaluation: LISS Survey Question Answers - Longitudinal Internet studies for the Social Sciences (LISS) panel - 7500 individuals over the age of 16 F-1 scores of our different models on two example LISS survey questions 3. Model evaluation F-1 scores of our different models on two example LISS survey questions Model Income R2 Demographic F1 Survey F1 Baseline 0.30 0.33 0.41 Pretrain + Static 0.50 0.32 0.64 Pretrain + Finetune 0.58 0.39 0.57 5. Interesting stuff: Does it make sense? Two-dimensional t-SNE projection of 2020 embeddings binned into hexagons of more than at least 10 persons. The points are coloured by the birth decade. 5. Interesting stuff: Does it make sense? Two-dimensional t-SNE projection of 2020 embeddings binned into hexagons of more than at least 10 persons. The points are coloured by the highest degree achieved (higher numbers indicate higher degrees). 5. Interesting stuff: Predicting further ahead -Embedding’s performance in longer-term prediction deteriorates less quickly compared to the linear model baseline F-1 scores of our different models on the marriage/divorce prediction https://www.flaticon.com/ Icons by Kiranshastry 5. Interesting stuff: Model size - Bigger != better! - Especially with fine-tuning, smaller models perform better - Hard to find a single best solution Parameter size of the best-performing fine-tuned and static embedding models per task https://www.flaticon.com/ Icons by Kiranshastry Task Fine-tuned Static embeddings Fertility 80 80 Marriage 160 540 Divorce 80 80 Income 80 540 Owning a car 80 80 Belongingness: Turks 160 540 5. Where are we and where do we go from here? - Fine-tuning our foundation models delivers performance (almost) on-par with state-of-art - We port our model within days - But performance still varies by task - Performance deteriorates slower compared to a baseline - But what is a good, scalable baseline? - Missing data & measurement error https://www.flaticon.com/ Icons by Freepik 5. Where are we and where do we go from here? - Attention weighted static embeddings - Generative models? - During pre-training, a predictive model should learn from only past events - Predicting a wider range of outcomes and further into the future? - Better integration of graph and sequence embeddings 5. Where are we and where do we go from here? - Explore our embedding space to describe the Dutch society -Track people’s life courses through time - Predict life outcomes further out into the future - Use embeddings for information-retrieval-like tasks - Create counterfactual scenarios - Explore (un)predictability per social group Thank you! a.macano[email protected] Appendix: Graph Embeddings Appendix: Model pre-training −Forthe MLM task, we do the following: 1. Randomly mask out (i.e. introduce gaps) 24% of the tokens. 2. Randomly change 3% of the tokens to any other random token. 3. Randomly select 3% of the tokens and leave them unaltered. −For the SOP task, the model has to predict which type of alteration has been done on each life-sequence: 1. 5% of life sequences are reversed. 2. The life-events are randomly shuffled in 5% of the life sequences. 3. The remaining 90% of sequences are left unaltered. −The final loss function is a weighted sum of the two objectives: L = 0.7 × MLM + 0.3 ×SOP