Dropout Prediction Using Advanced Machine Learning Models in a School and Community-Based Intervention to Promote Healthy Lifestyle and Prevent Type 2 Diabetes: Feel4Diabetes

Dropout Prediction Using Advanced Machine Learning Models in a School and Community-Based Intervention to Promote Healthy Lifestyle and Prevent Type 2 Diabetes: Feel4Diabetes Christos Tziastas1, Andreas Triantafyllidis1, Anastasios Alexiadis1, Konstantinos Votis1, Greet Cardon2, Jaana Lindström3, Violeta Iotova4, Imre Rurik5, Luis A. Moreno6,7,8,9, Eva Karaglani10, Christina Mavrogianni10, Yannis Manios10,11 1Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece 2Department of Movement and Sports Sciences, Ghent University, Ghent, Belgium 3Population Health Unit, Finnish Institute for Health and Welfare, Helsinki, Finland. 4Department of Social Medicine and Health Care Organization Medical University of Varna, Varna, Bulgaria 5Semmelweis University, Department of Family Medicine, Budapest 6Growth, Exercise, Nutrition and Development (GENUD) Research Group, University of Zaragoza, Zaragoza, Spain 7Centro de Investigación Biomédica en Red de Fisiopatología de la Obesidad y Nutrición (CIBERObn), Instituto de Salud Carlos III, Madrid, Spain 8Instituto Agroalimentario de Aragón (IA2), Zaragoza, Spain 9Instituto de Investigación Sanitaria de Aragón (IIS Aragón), Zaragoza, Spain 10Department of Nutrition and Dietetics, School of Health Science & Education, Harokopio University, Athens, Greece 11Institute of Agri-food and Life Sciences, Hellenic Mediterranean University Research Centre, Heraklion, Greece 2 Abstract. Participant dropout from interventional studies targeting healthy lifestyles can significantly undermine the validity of study outcomes. Accurate dropout prediction can help mitigate this issue by enabling proactive participant engagement strategies. This study aims to develop a robust Machine Learning (ML) model to predict dropout from a school and community-based interventional study to promote a healthy lifestyle and prevent type 2 diabetes: The Feel4Diabetes study. Using data from 3274 participants across 790 variables, we aim to identify key dropout determinants and enhance ML predictive accuracy. We evaluated three individual machine learning models—Random Forest, XGBoost, and Support Vector Machine (SVM)—based on performance metrics including accuracy, precision, recall, and F1-score. Among these, the Random Forest model emerged as the most effective, achieving an accuracy of 0.80 on the test set, with balanced precision and recall scores. Our study highlights the effectiveness of machine learning methods in predicting dropout in interventional studies promoting healthy lifestyles and preventing type 2 diabetes. Future research will concentrate on refining these models further and exploring additional data sources to enhance their generalizability. Keywords: Dropout Prediction, Machine Learning, Feature Selection, Type 2 Diabetes Prevention 1 Introduction 1.1 Background Chronic illnesses such as diabetes, heart disease, and cancer are primary causes of premature mortality, accounting for over two-thirds of all deaths and a substantial portion of healthcare expenditures. In Europe, these conditions consume an estimated 75% of healthcare budgets [1]. According to the World Health Organization (WHO), the global proportion of deaths due to chronic illnesses is projected to rise from 57% to 65% by 2030 [1], underscoring the urgent need for effective interventions. However, high dropout rates pose a significant challenge to the efficacy of health interventions. It is crucial to highlight and analyze usage metrics and determinants of attrition to understand how these applications perform among users who continue to engage with them [2]. Adherence is a major issue in health promotion programs, with many participants discontinuing use before completing the intervention. This phenomenon, known as non-usage attrition or dropout, can drastically affect the program's effectiveness [3]. High dropout rates not only prevent participants from receiving the full benefits of the intervention but also lead to inefficient use of resources, increased costs and introduction of bias in the results [4]. Identifying predictors of dropout has been the focus of several studies, though no consistent set of predictors has been established [5,6,7,8]. High dropout rates from health promotion interventions highlight the need for effective predictive models to identify at-risk participants and enable timely interventions to improve retention [8]. 3 Predictive modeling techniques like survival analysis, logistic regression, and random forests have been applied to predict dropout in educational settings, facing similar issues [9,10]. However, the application of these techniques in health interventional studies settings is limited, suggesting a gap in integrating predictive analytics into healthcare interventions. This study aims to develop an advanced machine learning (ML) pipeline to predict dropout from the Feel4Diabetes study, a school and community-based intervention designed to promote healthy lifestyles and prevent type 2 diabetes. By analyzing data from 3274 participants across 790 variables, we aim to identify key determinants of dropout and enhance the predictive accuracy of ML models. Our approach includes rigorous data preprocessing, feature selection using Sequential Backward Floating Selection (SBFS), addressing class imbalance with the Synthetic Minority Oversampling Technique (SMOTE), and hyperparameter tuning via GridSearchCV for Random Forest, Support Vector Machine and XGBoost models (Figure 1). We then select the best performing model to achieve superior predictive performance. 1.2 Materials and Methods The Feel4Diabetes dataset is a comprehensive repository containing data from 3,274 participants across 790 variables. The dataset comprises extensive participant information, including demographic, clinical, and behavioral attributes, which are essential for developing accurate predictive models. Through rigorous preprocessing, feature selection, and model tuning processes, we aim to identify key determinants of participant dropout and enhance the predictive accuracy of our models. 4 Figure 1:Block diagram of the dropout prediction pipeline 1.3. Preprocessing Effective machine learning models are built on well-prepared datasets. In this chapter, we focus on the preprocessing of the feel4diabetes.csv dataset, which is crucial for accurately predicting participant dropout in diabetes follow-up studies. In this proposed framework, the preprocessing steps included removing rows with more than 12% missing values, because we wanted to keep some features that had a bit more than 10% missing values such as encouragement to walk/bicycle and fruits vegetables intake as they showed correlation to the dropout. This ensures that our machine learning models are based on the most reliable information. In preparing the dataset for predicting participant dropout, we first excluded columns irrelevant to our analysis, such as 'center', 'schoolcode', 'classcode', and various demographic details, so as to focus on the most influential variables. Simultaneously, we assessed the dataset for missing data, sorting the columns by missing data percentages in ascending order. This step was crucial for identifying and prioritizing features for our analysis. We then embarked on an iterative process to refine 5 our feature set. Our approach was to maintain a balance between data completeness and feature richness, selecting features with minimal missing values and conditionally adding those with less than 12% missing data. This strategy yielded a dataset which consisted of 15 features from the initial 790. We also constructed a correlation heatmap of the features to display the correlation between multiple variables (Figure 2) and Violin Plots for comparing the probability distributions across different predictive features (Figure 3) Figure 2:Correlation heatmap of selected features 6 Figure 3:Violin Plots of predictive features 1.4. Sequential Backward Floating Selection After the preprocessing stage, we employed the Sequential Backward Floating Selection (SBFS) algorithm to extract meaningful features for training the model. Feature selection is crucial for enhancing the robustness and efficiency of machine learning models, particularly in datasets with a large number of features like Feel4diabetes. Among various heuristic approaches, we chose the SBFS algorithm for its effectiveness in identifying the most predictive features. The SBFS algorithm begins with the complete set of features and iteratively removes the feature whose exclusion maximizes classifier performance. Following each removal, it conditionally reintroduces features that could enhance the classifier's 7 performance. This alternating process of exclusion and conditional inclusion continues until the desired number of features is achieved, resulting in a subset of highly predictive features[11].In our dataset, the SBFS algorithm reduced the dimension of the feature space to 10 features (from 15 that had been selected in the preprocessing step).The features that were deemed the most influential for predicting the dropout were :Smoking status, Body-mass index (BMI), Waist circumference, FindRisk score, Physical Activity, Fruit/vegetable intake, Blood Glucose, Diabetes History,Age and Encouragement to walk/Bicycle. 2.SMOTE SMOTE (Synthetic Minority Over-sampling Technique) is an over-sampling method that generates synthetic examples to balance class distribution. The process involves creating synthetic samples along the line segments connecting each minority class sample to its k-nearest minority neighbors. For instance, with five nearest neighbors and a 200% over-sampling requirement, two neighbors are chosen, and synthetic samples are generated between each pair. This is done by adding a random fraction of the difference between a sample and its neighbor to the sample. This method effectively generalizes the decision region for the minority class, improving model performance. We applied the SMOTE in the training set after splitting to make sure that the model will be evaluated in original data and ensure robustness (Figure 4). Figure 4:Class distribution before and after SMOTE 8 1.2 Hyperparameter tuning – K fold cross validation Following the balancing of the dataset using SMOTE, we performed hyperparameter tuning through GridSearchCV. This approach efficiently searches the hyperparameter space, aiming to find the optimal parameters. We implemented 5-fold cross-validation to ensure robustness and reliability of the model evaluation. The data was split into training and testing sets with an 80-20 ratio. This demonstrated that the chosen hyperparameters were effective in enhancing the model's performance. 3.ML Models training We explored three machine learning models—Random Forest, Support Vector Machines (SVM) and XGBoost to predict dropout from the Feel4Diabetes interventional study. Each model underwent hyperparameter tuning using the previously described GridSearchCV method combined with 5-fold cross-validation. Definition 1.1. A random forest is a classifier consisting of a collection of treestructured classifiers {h(x,Θk ), k = 1, . . .} where the {Θk} are independent identically distributed random vectors and each tree casts a unit vote for the most popular class at input x.[12] Definition 1.2. Support vector machines (SVMs) are particular linear classifiers which are based on the margin maximization principle. They perform structural risk minimization, which improves the complexity of the classifier with the aim of achieving excellent generalization performance. The SVM accomplishes the classification task by constructing, in a higher dimensional space, the hyperplane that optimally separates the data into two categories.[13] Definition 1.3. XGBoost stands for Extreme Gradient Boosting, which applies a Gradient Boosting technique based on decision trees. It constructs short, basic decision trees iteratively. Each tree is termed as a “weak learner” because of its high bias. XGBoost begins by building the first basic tree that has a poor performance. Then it builds another tree, trained to predict what the first tree, which is a weak learner, cannot do. The technique sequentially produces weaker learners, each correcting the previous tree before the stopping condition is met, such as the number of trees (estimators) to be created [14]. 9 To enhance the performance of our machine learning models, we focused on selecting the most effective classifier among the three. In this regard, we concentrated on tuning and evaluating individual models to identify the one with the best predictive performance. After tuning each model—Random Forest, XGBoost, and Support Vector Machine (SVM)—we thoroughly assessed their performance metrics, including accuracy, precision, recall, and F1-score, on the test set. This rigorous evaluation process was aimed at determining which model provided the highest accuracy and most reliable predictions. Among the evaluated models, the Random Forest model emerged as the best performer, demonstrating superior accuracy, precision, recall, and F1-score compared to XGBoost and SVM. This approach of selecting the highest-performing individual model ensured robust and accurate predictions, making it the preferred choice for our study. The results validated the effectiveness of this strategy, confirming that carefully choosing and optimizing the best-performing single model can achieve high predictive performance. These findings are critical for enhancing dropout prediction in interventional studies, such as the Feel4Diabetes study, aimed at promoting healthy lifestyles and preventing type 2 diabetes. Future efforts will focus on further refining the Random Forest model and incorporating additional data sources to improve its generalizability, providing valuable insights for researchers and practitioners working to enhance participant retention in similar studies. 5.Results The integration of SMOTE for data balancing, k-fold cross-validation with hyperparameter tuning, and Sequential Backward Floating Selection (SBFS), followed by a selection of the best individual model, resulted in improved accuracy for predicting dropout from the feel4diabetes study in comparison to each individual model. The Random Forest at 0.80, XGBoost at 0.77, SVM at 0.78. This underscores the Random Forest’s accuracy and overall performance compared to the SVM and XGBoost. To highlight this, we constructed the ROC-AUC curve of the Random Forest (Figure 6) and the performance metrics of each respective model (Figure 5).