Text-based approach for detecting cases of ADEs from EHRs of participants of the Estonian Biobank

Contents lists available at ScienceDirect Informatics in Medicine Unlocked journal homepage: www.elsevier.com/locate/imu Text-based approach for detecting cases of ADEs from EHRs of participants of the Estonian Biobank Dage Särga,b,∗ , Kairit Sirtsb, Kristi Krebsa, Markus Tammd, Alise Metskülad, Marek Ojab, Sven Laurb, Jaak Vilob,c, Lili Milania aEstonian Genome Centre, Institute of Genomics, University of Tartu, Tartu, Estonia bInstitute of Computer Science, University of Tartu, Tartu, Estonia cSTACC, Tartu, Estonia dFaculty of Medicine, University of Tartu, Tartu, Estonia A R T I C L E I N F O Dataset link:https://genomics.ut.ee/en/conten t/estonian-biobank Keywords: Adverse drug events Electronic health records Natural language processing Machine learning A B S T R A C T Objective: This study develops methods for detecting Adverse Drug Events (ADEs) from Electronic Health Records (EHR) of participants of the Estonian Biobank to support pharmacogenetics research. It focuses on creating manually annotated datasets and improving ADE detection efficiency by combining rule-based and machine learning (ML) approaches, specifically applied to antidepressants and antipsychotics. Materials and Methods: To detect potential ADE mentions within free text fields of EHRs, we employed a lexicon-based approach to extract text snippets containing both a drug name and a symptom. We developed a rule-based and an ML-based system to prefilter the extracted text snippets, aiming to reduce the number of non-ADE snippets going into manual annotation. We then applied both systems before manual annotation and assessed their impact on the annotation process. Results We produced annotated datasets for antidepressants (520 patient–drug pairs) and antipsychotics (1,329 pairs). Our prefiltering method reduced the annotation workload up to 24-fold compared to no filtering, and ML-based filtering outperformed rule-based filtering, requiring only 1.3–1.5 snippets per positive ADE case. Pharmacogenetic validation revealed significant genotype — ADE associations for Escitalopram, Sertraline, and Quetiapine. Conclusion The implementation of prefiltering methods significantly enhanced the efficiency of manual annotation for dataset creation and pharmacogenetic validation confirmed the datasets’ utility and usability. Therefore, we showed that ADE extraction from free text adds value by expanding the scope and diversity of analyzable cases for discoveries in the genetics of drug response. 1. Introduction Adverse Drug Events (ADEs) are a significant public health concern due to their detrimental impact on many individuals and their associated healthcare costs. ADEs constitute a leading cause of mortality and hospitalization in developed nations. An epidemiological review study has suggested that in the EU, approximately 0.25% (or 1 in 400 hospitalized patients) of all patients who are not hospitalized due to an ADE will die as a result of an ADE during their stay in hospital [1]. About 6.5%–15% of hospital admissions in adults are caused by ADEs [2,3], and 15% of hospitalized patients experience ADEs [4]. Previous research has demonstrated that certain ADEs may be linked to an individual’s genotype. For example, extensive studies ∗Correspondence to: Estonian Genome Centre, Institute of Genomics, University of Tartu, Riia 23b/2, 51010 Tartu, Estonia. E-mail address: [email protected] (D. Särg). have explored the relationship between a person’s genetics and their antidepressant response [5–7]. Such findings have the potential to inform the development of personalized medicine. To investigate the relationship between genetic variation and ADEs, a dataset containing information about individuals’ genomes, the drugs they have taken, and their response to these drugs is required. One such dataset, utilized in this study, is the Estonian Biobank (EstBB) [8,9] which includes genomic data and electronic health records (EHRs) for more than 200,000 individuals, accounting for approximately 20% of the adult population of Estonia. This biobank has been extensively used in various studies, as detailed in [9,10]. However, in EHRs, ADEs are typically recorded only in free text format within the anamnesis or summary fields. This presents a significant challenge for retrieving https://doi.org/10.1016/j.imu.2025.101701 Received 6 May 2025; Received in revised form 23 September 2025; Accepted 8 October 2025 Informatics in Medicine Unlocked 59 (2025) 101701 Available online 10 November 2025 2352-9148/© 2025 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). This version of the article has been accepted for publication, after peer review, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1016/j.imu.2025.101701. D. Särg et al. information on ADE cases, as manual review of millions of documents is not feasible for large-scale studies. Therefore, computational methods are necessary to address this issue. Recent advances in language models have transformed natural language processing in many fields, including healthcare. In the past five years, research in the field of clinical natural language processing (NLP) for the extraction of ADEs from free text health records has gained considerable ground. [11] published a scoping review on text-based ADE detection from clinical data. They found that existing research predominantly focused on two publicly available pre-annotated Englishlanguage datasets: MADE 1.0[12] and n2c2[13]. The MADE 1.0 dataset includes the EHRs of 21 randomly selected cancer patients, while the n2c2 consists of 505 discharge summaries selected from the MIMIC III dataset [14] using a query that searched for an ADE in the International Classification of Diseases (ICD) code description of each record [13]. Transformer-based architectures such as BERT and GPT variants have demonstrated superior performance over traditional rule-based and machine learning methods in extracting information from clinical narratives. This has been demonstrated in several works, e.g. [15–18]. There is considerably less research on how the methods developed based on the two datasets perform on real-world clinical data to support research in other areas, such as pharmacogenetics or pharmacovigilance. A scoping review on NLP and ML methods for ADE detection from EHR-s [19] published in 2025 identified only seven studies that met their criteria of being used directly for pharmacovigilance. One of the studies identified was published in 2023 [20] and used keyword search for text mining. The rest were published in 2020 and earlier, highlighting the scarcity of recent high-impact work in this domain. At the same time, [21] highlight that NLP methods have the potential to provide enormous benefit in treatment optimization. Information retrieved from the free text fields of EHRs could be used to build models for predicting drug metabolism and their optimal dosing and to link these models to real-world outcomes [21]. Narrative reviews further highlight the potential of transformer systems for pharmacovigilance and clinical decision support, while noting challenges such as interpretability, domain-specific variability, and privacy constraints [22]. These developments underscore the growing relevance of NLP-based approaches and motivate future work to adapt such models for low-resource languages and sensitive clinical data, as explored in our study. Our work contributes to the field by bringing state-of-the-art natural language processing methods to use in pharmacogenomics. This paper introduces a hybrid approach combining rule-based and transformerbased ML methods to prefilter EHR text snippets, significantly enhancing the efficiency of manual annotation of ADEs. First, focusing on antidepressants (AD), we developed a rule-based filtering method to select data for annotation. Next, we explored how combining the rule-based method with a supervised ML model trained on the AD dataset could help to pre-select data for annotation in a slightly different domain — antipsychotic (AP) drugs. Finally, we evaluated how different combinations of inand out-of-domain data (i.e. AD vs AP when evaluated on AP) influence the model’s performance, robustness, and generalizability. This helps determine whether the model can benefit from additional data, even if it is not perfectly aligned with the target domain. Overall, we found that even the out-of-domain model outperformed the rule-based approach, and that the model trained on the largest dataset, including both inand out-of-domain data, achieved the highest F1-score, demonstrating the importance of extensive training data. Including in-domain samples in the training data significantly improved recall which underscores the value of domain-specific data in enhancing model performance. We validated the approach through an experiment replicating known pharmacogenetic associations on the extracted cases. Further, our approach was used to identify additional cases in [23]. To our knowledge, this is the first study applying transformer-based NLP methods to support pharmacogenomic research by extracting ADE cases from EHRs. Statement of significance Problem. Identifying individuals that have developed an adverse event from any certain drug so that associations between pharmacogenetic phenotypes and ADEs could be studied. What is already known? ADEs are often recorded in free text within EHRs. To identify which individuals have developed an ADE by manually reading through all the health records is laboursome and impractical. Most works focusing on automating this process concentrate on specially prepared de-identified datasets instead of actual clinical data, and are not directly transferable. There are no previous works that would attempt the task for EstBB. What this paper adds? This paper introduces a hybrid approach combining rule-based and ML methods to prefilter EHR text snippets, significantly enhancing the efficiency of manual annotation of ADEs. It also provides manually annotated datasets for antidepressants and antipsychotics, validated through pharmacogenetic associations. Who would benefit from the new knowledge in this paper? Researchers in pharmacogenetics, clinicians interested in ADE detection, and developers of EHR systems aiming to improve ADE detection efficiency. 2. Materials and methods 2.1. Source data — estonian biobank Our research is based on free text fields of health records of participants of EstBB. While the biobank currently includes data for over 200,000 individuals, these participants were recruited in multiple phases. Much of the work described in this paper was conducted when data were available for approximately 50,000 individuals. The EHRs of the additional 150,000 individuals recruited later were only incorporated in the final steps of evaluation which focused on the antipsychotics ADE dataset. The EHR data are initially received by EstBB in XML format. Each XML file contains multiple sections, where each section consists of a header and its corresponding content. These XML files are then parsed into a PostgreSQL database, preserving the section headers and their text content. For free-text sections, such as anamnesis and summary, an additional rule-based algorithm is applied after XML parsing to segment treatment events based on headers that include the visit date. The EHR processing pipeline is described in detail by [24]. 2.2. ADE dataset creation In this study, we created annotated datasets for two drug groups: antidepressants and antipsychotics. For both of these, similar methodology was employed that is described in the following subsections. Given the absence of a pre-existing ADE dataset for Estonian, we initiated a rule-based prefiltering approach to obtain an enriched sample of data for manual annotation. 2.2.1. Building lexicons: drug names and ADE symptoms All our dataset creation experiments started with a similar first step: lexicon-based detection of drug names and symptoms from EHRs. To construct comprehensive lexicons of drug names, we used the drug database of the Estonian Agency of Medicines,1 incorporating all drug and substance names, and accounting for variations in spelling. To compile a list of possible ADE symptoms, we used patient information leaflets (PILs). The ADE sections were extracted and segmented into individual words and phrases, considering both document structure and linguistic form. We mapped both the found drug names and ADE words and phrases on free text fields of the EHR-s, using the EstNLTK toolkit [25]. Further details on the lexicons can be found in Appendix A. 1https://ravimiregister.ee/?pv=PublicDownloads Informatics in Medicine Unlocked 59 (2025) 101701 2 D. Särg et al. 2.2.2. Preliminary approach In a preliminary experiment, we looked at the context of a sentence — if a sentence contained both a drug and a possible ADE symptom, this sentence together with its neighboring ones was picked for manual annotation. We annotated 11,179 sentence triplets and, from this effort, only 229 confirmed ADE cases were identified, highlighting the fact that drug-symptom co-occurrences in the same sentence rarely indicate an ADE. Furthermore, the wide range of diagnoses and drugs among patients rendered the data unsuitable for pharmacogenetic studies. To address these issues, we observed the need for a more specific approach and an additional data filtering step. We decided to focus on adverse events related to two specific drug groups: antidepressants (AD) and antipsychotics (AP). These drugs are frequently prescribed, promising a sufficient number of cases for downstream pharmacogenetic analysis. 2.2.3. Limiting the data pool To detect text snippets containing drug names and symptoms, we employed the lexicon-based approach described in Section 2.2.1, restricting the dataset to drug names within the relevant drug class (antidepressants) and symptoms extracted from the PILs of these drugs. Rather than considering all EHRs, we focused exclusively on records with an ICD-10 diagnosis from the F group (Mental and Behavioral disorders). As a subsequent step, we applied a rule-based prefiltering method to obtain an enriched sample of text snippets for manual annotation. We developed four sets of rules, primarily based on regular expressions. These rule sets were formulated based on insights gained from the preliminary data annotation attempt, a manual review of health records with depression diagnosis, and consultations with a medical professional regarding the documentation of antidepressant ADEs in health records. A natural language description of the rules is provided in Appendix B. The rules were applied to text snippets extracted from the free text fields. These snippets contained a drug name or substance and a symptom within 300 characters of each other, including 100 characters before the first entity and 100 characters after the second entity. There were no restrictions on the order in which the drug and symptom appeared. In addition to the four sets of rules, we incorporated a rulebased negation detection system that accounted for various expressions of the absence of adverse events. 2.2.4. Manual annotation For manual annotation, the text snippets containing the drug and symptom (highlighted between special characters) were presented to the human annotator. The data was annotated by one expert annotator to determine whether the text snippet contained an adverse event and if the highlighted symptom resulted from the highlighted drug. Hence, the annotator classified the text snippets into 4 annotation categories: •0: the text snippet does not contain any ADE •1: the text snippet contains an ADE and it is the marked symptom resulting from the marked drug •2: the text snippet contains an ADE but it is not between the marked drug and/or symptom •3: there is not enough information/not possible to tell based on the given context A single annotator was deemed sufficient thanks to the task’s perceived simplicity and objectivity. To verify this simplicity, we had a subset of the data annotated by a non-expert and evaluated the inter-annotator agreement by calculating the Cohen kappa. More details on the manual annotation can be found in Appendix C . 2.3. Modeling We used two models in different stages of our study: EstMedBert [26] and EstRoberta.2 Both models leverage the transformer architecture, enabling them to understand and generate human-like text by capturing contextual relationships between words. EstMedBERT is a BERT-based transformer model pre-trained specifically on Estonian EHR texts. In contrast, EstRoBERTa is pre-trained on a diverse corpus of Estonian texts, not limited to EHRs. Consequently, both models possess an inherent understanding of the Estonian language prior to fine-tuning with our data. Initially, EstMedBERT was our model of choice due to its domainspecific pre-training. However, during the course of our work, we transitioned to the EstRoBERTa model, as it significantly outperformed EstMedBERT on the AD data, achieving an F1 score of 0.80 compared to EstMedBert’s 0.68. This performance disparity can be attributed to two main factors. Firstly, the RoBERTa (Robustly optimized BERT approach) architecture incorporates several enhancements over the BERT model, making it generally more effective and robust. Secondly, our task of relationship classification is not strictly confined to medical terminology and language, allowing EstRoBERTa to benefit from its broader pre-training data. We fine-tuned both models for binary classification, for which we converted the annotated datasets into having just two labels: positives (true ADE between the highlighted drug and symptom) and negatives. Therefore, from our manual annotation labels, we used annotation category 1 as positives, combined annotation categories 0 and 2 as negatives, and left out category 3. More details, including hyperparameters used for fine-tuning, can be seen in Appendix D. 2.4. Study process Our work flow is illustrated in Fig. 1. Fig. 1(a) shows the three consecutive stages related to the methodology for the generation of datasets. Fig. 1(b) depicts our pharmacogenetic validation process. The three stages, as well as the validation process are described in the following subsections. 2.4.1. Stage 1: Rule-based approach for the extraction of the antidepressant ADE dataset In the first stage, to construct a manually annotated dataset of ADEs from antidepressants, we used the methodology outlined in Section 2.2. This way we received the AD dataset that we could use in the next stage of our study. 2.4.2. Stage 2: Hybrid approach for antipsychotics ADE dataset In the second stage, we created a dataset of ADEs from antipsychotics. The annotation was performed in two rounds, as we incorporated an ML model in addition to the rules. In the first round of annotation, we created one batch of training data (AP1) and a development set, in the second round we created one more batch of training data (AP2). For the first round, we fine-tuned EstMedBert [26] for text classification on the annotated AD data (referred to as the ‘‘AD model’’ in Fig. 1(a)). For details on data augmentation and fine-tuning, please refer to Appendix D. The F1 score of the model on AD test set was 0.68, with precision at 0.72 and recall 0.65. While this is not good enough to be used on its own for ADE detection, we hypothesized it would help us with additional filtering for the data to be manually reviewed. Hence, in addition to the rules, we applied the EstMedBert AD model on the AP text snippets, referred to as ‘‘hybrid relation detection’’ in Fig. 1(a). For the training batch annotation, we preferred text snippets with a higher probability of being a positive example according to the regexes 2https://huggingface.co/EMBEDDIA/est-roberta Informatics in Medicine Unlocked 59 (2025) 101701 3 D. Särg et al. (a) Dataset Development (b) Pharmacogenetic Validation Fig. 1. Study Process. and/or AD model. We annotated 390 examples resulting in 349 indomain training examples for AP drugs (for 41 examples, it was not possible to say whether this is an ADE or not), 231 positive and 118 negative. To avoid the data scarcity problem in the final dataset as seen with AD-s, we only included examples of the three most commonly prescribed AP-s in EstBB: Quetiapine, Olanzapine, and Clozapine. We only took the highest-confidence example from each patient to prevent data leak between training and evaluation data and to discover as many patients with ADEs from AP-s as possible. See more details in Appendix C. For the development set, we grouped text snippets into 50 classes based on 5 rule types (including one for uncaptured snippets) and 10 model confidence score bins. We then randomly selected up to 5 examples from each class. This approach captures the diversity of rule types and confidence levels, addresses dataset imbalance by including an equal number of positive cases, and improves the robustness of our evaluation. In total, we annotated 220 text snippets for the development set, 106 of them positive, 89 negative and 25 left out as unclear. After the first annotation round, we retrained the EstMedBert model using the AP1 set in addition to the AD dataset, by concatenating the two datasets. For the second annotation round (AP2 training batch), we chose text snippets concerning AP drugs other than the three in the AP1 set. For manual annotation, we picked 390 snippets for which the retrained model’s confidence score was > 0.5. The annotation resulted in 330 new training examples, 139 positive and 191 negative. 2.4.3. Stage 3: Modeling in-domain vs out-of-domain In Stage 3, we assessed how incorporating different types of training data influences the performance of a machine learning model on the AP ADE detection task. In-domain data pertains directly to the specific task, here AP ADEs, whereas out-of-domain data includes related but not directly relevant information, here AD ADEs. As stated in Section 2.3, in this stage we used the EstRoberta model. The datasets for AD and AP also differed in terms of data (im) balance. The training set sizes, as well as the ratios of positive vs negative examples can be seen in Table 1. The AD dataset was significantly larger but heavily imbalanced, with only 17% positive examples. Conversely, the AP datasets had a higher proportion of positive examples, Table 1 Training datasets: overview. Training data Total Positive Negative AD 4078 694 (17%) 3384 AP1 349 231 (66%) 118 AP2 330 139 (42%) 191 66% and 42% for AP1 and AP2, respectively. Therefore, while the AD training set is about 6 times the volume of both AP sets together, the difference in the counts of positive examples is less than 2-fold. At Stage 3 of our study, where we needed to evaluate the methodology on a hold-out test set of antipsychotics to compare the performance of models trained on different data subsets, we had gained access to the EHRs of an additional 150,000 participants of EstBB. Therefore, for the test set we had a considerably larger data pool than for the previous annotation rounds. For testset annotation, we ran the model trained on AD and AP1 data on the additional EHR-s and had 1004 text snippets annotated. To balance the dataset, we deliberately included 250 random examples that the model predicted to be negative. The final size of the hold-out set was 960 text snippets, 464 positive and 496 negative (44 snippets were left out as unclear). 2.4.4. Pharmacogenetic validation To verify the usability of the created dataset, we studied the association between ADEs and pharmacogenetic phenotypes for Escitalopram, Sertraline, and Quetiapine, focusing on the CYP2C19 and CYP2D6 genes. To define controls, we analyzed digital drug dispensing data from EstBB. Controls had at least three drug dispenses with no more than a 90-day gap between them, and used two or fewer different drugs. Using PharmCAT [27] and Stargazer [28] algorithms, we categorized individuals into metabolizer subgroups and compared ADE reporting. Logistic regression, adjusted for sex, birth year, and population structure, was performed using R software, excluding relatives. The methodology of the analysis is further explained in Appendix E. Informatics in Medicine Unlocked 59 (2025) 101701 4 D. Särg et al. Table 2 Performance metrics on the manually annotated development set. Combined 0 - if one system predicted negative, it was considered a negative. Combined 1 - the other way round. Rules Model Combined 0 Combined 1 Precision 0.54 0.75 0.70 0.59 Recall 0.67 0.64 0.41 0.91 F1 0.60 0.69 0.52 0.72 (tp+fp)/tp 1.86 1.34 1.42 1.69 3. Results 3.1. Stage 1: Rule-based approach for antidepressant ADE dataset In total, we extracted 3402 text snippets with possible mentions of antidepressant ADEs, pertaining to 2072 distinct patients. The annotator manually reviewed 1189 of these text snippets, which included data for 908 patients (1052 distinct patient-drug pairs). Among these, ADEs were confirmed for 445 patients (520 patient–drug pairs) by the annotator. This indicates that to obtain 520 positive cases, 1189 text snippets needed to be annotated, resulting in an average of 2.3 snippets per positive example. In comparison, our preliminary study required the review of 48.8 data points (sentence triplets) to retrieve a single positive case. Thus, the additional filtering rules enhanced the efficiency of the annotation process by more than 20-fold, making dataset generation significantly more feasible. Based on 200 random examples that were doubly annotated by an expert and a non-expert, Cohen’s kappa was 0.83. According to the metric’s interpretation, this indicates a strong agreement, even when considering higher thresholds required for medical studies [29]. Our final annotated dataset comprised 520 cases of individuals experiencing ADEs from antidepressants, encompassing 16 different substances. The most common substances were escitalopram (153 patients), sertraline (108 patients), and mirtazapine (47 patients), reflecting the overall frequency of prescription of these drugs in EstBB. 3.2. Stage 2: Hybrid approach for antipsychotics ADE dataset For AP dataset, we evaluated the performance of the rules and the model both separately and combined. The results listed in Table 2 indicate that introducing the model to the process improves both precision and recall. Precision is highest using only the model; compared to the rules-only system, the precision is 0.21 higher while we lose less than 0.03 in recall. For recall optimization, the Combined 1 system where a prediction is considered positive if either the model or the rules predict it to be positive, is the best performer. This combination’s recall is 0.24 higher and precision 0.05 higher than in the baseline rules-only system. The last row in the table shows that, for annotation efficiency, using the model only is optimal, but both combined systems are also more efficient than rules only. Based on 200 random doubly annotated text snippets, the Cohen kappa in this experiment was 0.89. After combining the positively annotated examples from all batches, our pharmacogenetic AP dataset included 1329 individuals. The most common AP substances causing an ADE in our dataset were quetiapine (424 patients), olanzapine (194 patients), aripiprazole (70 patients), clozapine (66 patients), and risperidone (60 patients). 3.3. Stage 3: Modeling: in-domain vs out-of-domain To optimize the model training, we explored combined datasets for the training of the model. Table 3 lists the data combinations we used, and presents performances of the models on the AP dataset. We observed that the model fine-tuned on the largest amount of data (AD+AP1+AP2) has the highest F1 score of 0.77. The lowest performers are the models trained on the least amount of data — the AD model Table 3 Performance metrics on AP test data (n=960, pos = 464, neg = 496) Training data F1 Precision Recall (tp+fp)/tp AD 0.71 0.70 0.71 1.43 AD + AP1 0.76 0.67 0.87 1.50 AD + AP2 0.73 0.76 0.70 1.32 AD + AP1 + AP2 0.77 0.69 0.88 1.45 AP1 + AP2 0.70 0.55 0.95 1.82 Fig. 2. Precision–Recall curves of models trained on different batches of data. trained only on out-of-domain data and the AP1+AP2 model trained only on small amounts of in-domain data. While their F1 scores are similar, the AD model’s precision and recall are almost equal (0.01 difference) whereas the in-domain model has a much higher recall (0.95), the highest of all the models. Its precision, however, is the lowest at 0.55. Based on these results, we can conclude that including in-domain samples in the training data is beneficial for the model performance, first and foremost in terms of recall. Compared to a model trained only on out-of-domain data, the gain in recall is 0.17. This comes at the price of losing 0.01 in precision. When using only a small sample of indomain data without any out-of-domain data, the recall is even higher but the loss of 0.15 in precision does not make this a viable option. The precision–recall curves of the models can be seen from Fig. 2. It shows that the AD+AP1+AP2 model marked by the yellow line has the highest F1-score throughout the whole curve. The highest precision is achieved at a fairly low recall score of 0.3–0.4, however, there are no steep declines in precision when moving towards higher recall rates. 3.4. Pharmacogenetic validation In a logistic regression analysis between ADEs reported from Escitalopram we validated previous findings of [30,31] that individuals with the CYP2C19 PM phenotype were more likely to report ADEs from Escitalopram compared to normal metabolizers (NM; OR = 2.39, 95% CI 1.21–4.71). Further, we see that the CYP2C19 diplotype *2/*2, which indicates a complete loss of function, was associated with reporting ADEs from Escitalopram (OR = 2.26, 95% CI 1.10–4.64). In analysis between ADEs reported from Sertraline, we likely lack sufficient power, which may explain the insignificant association observed with the CYP2C19 poor metabolizer (PM) phenotype known to influence sertraline exposure [32]. However, we did detect the expected association with CYP2C19 diplotype *1/*2 (OR = 1.72, 95% CI 1.05–2.81), see Fig. 3 in Appendix E. This diplotype indicates an intermediate metabolizer phenotype of the CYP2C19 enzyme which plays a critical role in Sertraline metabolism. For the antipsychotic Quetiapine, we examined its association with the CYP2D6 enzyme, which has a secondary role in Quetiapine Informatics in Medicine Unlocked 59 (2025) 101701 5 D. Särg et al. metabolism [33]. We found that individuals with the CYP2D6 PM phenotype were significantly more likely to report ADEs from Quetiapine compared to NMs (OR = 2.08, 95% CI 1.24–3.45). Significant association was also observed in individuals carrying two non-functional CYP2D6 alleles (*4/*4 diplotype, OR = 2.24, 95% CI 1.19–4.20). Supporting these findings, a two-patient case study previously reported severe ADEs in carriers of the CYP2D6*4 variant when treated with Quetiapine [34]. These findings underscore the potential of mining free text fields of medical records to identify ADE cases, further enabling analysis of associations with pharmacogenetic phenotypes. 4. Discussion and future work 4.1. Dataset creation efficiency Our study demonstrates that automated filtering after lexicon-based annotation of drug names and symptoms in EHRs significantly improves the creation of datasets for pharmacogenetic studies. Initially, without pre-annotation filtering, we needed to manually annotate nearly 50 data points to find one positive example of an adverse drug event, requiring over 12,000 annotations for a dataset of 250 positive individuals. Incorporating rule-based filtering reduced the data needed for annotation 24-fold, making ca 500 snippets sufficient for the same dataset. Training an ML model to identify ADEs further improved accuracy, lowering the ratio to 1.3–1.5 snippets per positive example, thus requiring 325–375 annotations for 250 positive cases. However, focusing solely on precision is impractical due to the limited data pool, emphasizing the importance of in-domain training data. While the precision of the baseline (AD) model and the comprehensive (AD+AP1+AP2) model are similar, the latter’s recall is 0.17 higher, reducing undetected true positives from 29% to 12% with the same manual effort. 4.2. Errors analysis While additional filtering makes dataset creation more efficient, our findings demonstrate that fully automated case identification is not yet feasible. The highest precision we achieved was 0.76 by combining the AD and AP2 training sets. This still means that, without manual annotation, 24% of cases detected by the model would be false. In addition, its recall of 0.67 leaves a significant proportion of true ADE cases undetected, which reduces statistical power for association studies. As was shown in Table 3, the model trained on the biggest amount of data (AD+AP1+AP2) gave the highest F1 score with a considerably high recall of 0.88. However, its precision was 0.69, meaning that almost one third of the cases it detected as positive were false, and without manual review, the dataset would not be reliable for pharmacogenetic studies where high precision is critical. To understand which cases present problems for the model, we performed error analysis shown in Table 4, and an example of each category is presented in Appendix F. We reviewed 100 random false positive examples (FP) and all 43 false negatives (FN) of the AD+AP1+AP2 model. FPs were dominated by three main patterns. First, no causal relation between drug and symptom (29 cases), where the model misinterpreted co-occurring mentions as an ADE. Second, ADE attributed to a different drug (29 cases), reflecting confusion in multi-drug contexts when the adverse event was linked to another medication. Third, indication mistaken as ADE (25 cases), where the symptom was actually the reason for prescribing the drug rather than an adverse effect. Additional FP sources included hypothetical or educational context (10 cases), such as patient counseling or warnings about possible ADEs, annotation ambiguity (6 cases), where the interpretation was uncertain, and a single case of negation not detected. FNs revealed different challenges. The most frequent category was implicit ADEs signaled by treatment changes or dosage adjustments (16 cases), which require inference beyond explicit statements. Other FNs Table 4 Error analysis of the highest F1 score model (AD+AP1+AP2) Category Count False Positives (FP) No causal relation between drug and symptom 29 ADE attributed to a different drug 29 Indication mistaken as ADE 25 Hypothetical or educational context 10 Annotation ambiguity 6 Negation not detected 1 False Negatives (FN) Implicit ADE (treatment change or dosage adjustment) 16 Missed obvious ADE mention 13 Multiple drugs causing ambiguity 8 Drug and symptom in separate sentences 6 involved multiple drugs causing ambiguity (8 cases), where the correct pairing was unclear, and drug and symptom separated by sentence boundaries (6 cases), making it difficult for models to capture the relation. There were also several obvious ADE mentions (13 cases), indicating that even straightforward examples can be overlooked. These findings suggest that both causal reasoning and context interpretation remain key limitations of ADE detection. In addition, we looked at errors where none of the models got it right. There were 50 cases in the AP test set that all ML models misclassified as positives and 4 cases as negatives. The dominant categories among the FPs here were the hypothetical or educational context and ADE attributed to a different drug. We suppose that the first type may require more dedicated training data, while the second could benefit from a different mark-up scheme. 4.3. Dataset usability in pharmacogenetics Our methodology has already demonstrated practical utility in pharmacogenetic research as [23] studied antidepressant side effects in the Estonian Biobank. For the study, 1799 ADE cases identified using our ADE detection pipeline were successfully incorporated into association analyses alongside cases derived from structured questionnaires. While the questionnaire-based approach yielded a larger number of cases, it requires patient participation and additional data collection, and cannot be applied retrospectively. In contrast, our method enables large-scale mining of already existing clinical text, providing complementary and unique data for pharmacogenetic studies. This demonstrates that even in settings where structured data is available, NLP-based extraction adds value by expanding the scope and diversity of analyzable cases. 4.4. Limitations of the study As with any study, there are limitations to address. Recent Large Language Models (LLMs) such as ChatGPT have shown remarkable performance in natural language processing tasks, often outperforming older models. However, their performance on Estonian data is less researched. For instance, [35] showed that in stance detection, ChatGPT 3.5 performed on the same level as the EstRoberta model. This allows us to assume that newer LLMs, e.g. ChatGPT4, could also outperform EstRoberta in our task. Due to data protection and ethical requirements, we cannot test ChatGPT or other externally hosted models with our clinical notes, which contain sensitive patient information. Using inhouse models like Roberta allows us to maintain data control. While downloadable LLMs exist, their performance in Estonian is inferior to their success in widely used languages. Our study focused on antidepressants and antipsychotics, limiting generalizability to other drug classes. The symptom detection method based on PILs cannot identify unknown side effects, affecting the Informatics in Medicine Unlocked 59 (2025) 101701 6 D. Särg et al. system’s ability to detect ADEs expressed in complex linguistic constructions. Prefiltering may exclude positive examples, making it impossible to assess the recall loss compared to no prefiltering, as manual annotation without prefiltering is too laborious. 4.5. Conclusion In this study, we demonstrated that our hybrid methodology, combining rules and transformer-based models, significantly reduces the effort required to create a reliable ADE dataset from free text fields in health records. We developed and validated two annotated datasets, proving their usability in pharmacogenetics through logistic regression analysis. Using a machine learning model, even if trained on slightly out-of-domain data, was a more efficient approach for creating an annotated dataset than using the previously developed rules. However, there is still a lot of room for improvement both in terms of precision and recall. We optimized training strategy for an ML model tailored to this task. Our findings indicate that integrating in-domain and out-of-domain data enhances the model’s generalization capabilities, improving performance on unseen data which is especially useful in scenarios where large annotated datasets are scarce. While fully automating the verification process remains challenging, the datasets created in this study are invaluable for advancing evaluation and development efforts in both the specific task and the broader context of Estonian language and sensitive clinical data. Although large language models (LLMs) offer immense potential, our use case necessitates tailored solutions. Future research should focus on finetuning LLMs specifically for Estonian clinical text, bridging the gap between state-of-the-art models and language-specific requirements. CRediT authorship contribution statement Dage Särg: Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Formal analysis, Data curation, Conceptualization. Kairit Sirts: Writing – review & editing, Writing – original draft, Supervision, Methodology, Conceptualization. Kristi Krebs: Writing – review & editing, Writing – original draft, Visualization, Methodology, Formal analysis. Markus Tamm: Writing – review & editing, Data curation. Alise Metsküla: Writing – review & editing, Data curation. Marek Oja: Writing – review & editing, Resources, Data curation. Sven Laur: Writing – review & editing, Resources, Data curation. Jaak Vilo: Writing – review & editing, Resources, Funding acquisition. Lili Milani: Writing – review & editing, Supervision, Resources, Methodology, Funding acquisition, Conceptualization. Ethics The activities of EstBB are regulated by the Human Genes Research Act, which was adopted in 2000 specifically for the operations of EstBB. Individual level data analysis in EstBB was carried out under ethical approvals 1.1-12/624 and 1.1-12/3797 from the Estonian Committee on Bioethics and Human Research (Estonian Ministry of Social Affairs), using data according to release application 6-7/GI/22640 from the Estonian Biobank. Funding Funded by the European Union under the Horizon Europe Action Grant Programme agreement No 101057639 project SafePolyMed. Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. This project has received funding from the Swedish Research Council grant number 2021-02732 and Estonian Research Council grants PRG2625, PRG1844, and PSG721. Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Lili Milani reports financial support was provided by European Union. Lili Milani reports financial support was provided by Swedish Research Council. Kairit Sirts, Jaak Vilo, Lili Milani reports financial support was provided by Estonian Research Council. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments Data analysis was carried out in part in the High-Performance Computing Center of University of Tartu. We acknowledge the work of Estonian Biobank research team (Andres Metspalu, Tõnu Esko, Reedik Mägi, Mait Metspalu, Mari Nelis and Georgi Hudjashov), the Health Informatics Research team (Markus Haug, Raivo Kolde, Kerli Mooses, Maarja Pajusalu, Sulev Reisberg, Hendrik Šuvalov, Harry-Anton Talvik), as well as all staff at the Estonian Biobank and the participants of the biobank. The research was conducted using the Estonian Center of Genomics/Roadmap II funded by the Estonian Research Council (project number TT17). Microsoft Copilot was used for language correction assistance during the final preparation of this manuscript. Appendix A. Lexicon building details To compose a comprehensive lexicon of drug and substance names, we used the drug database from the Estonian Agency of Medicines.3 We extracted all the drug names and substances (in Estonian and in Latin). When mapping them on the free text fields, we also allowed an edit distance of 1–4 depending on the length of token and the specific differences between the tokens (e.g. letters ’s’ and ’z’ could be switched). This was done both based on our own observations as well as the information received from the Agency of Medicines concerning the constraints for naming new drugs that enter the Estonian market. The total size of the lexicons was 774 name variants for AD (13 substances, 61 drug names) and 1746 name variants for AP (19 substances, 82 drug names). To get a list of possible ADE symptoms, we used patient information leaflets which are semi-structured documents. We extracted the ADE sections and split them into separate words and phrases describing ADEs based on both the structure of the document (e.g. bulleted items) and the linguistic form of the sentences/bulleted list items (e.g., an ADE symptom should be a noun phrase). We mapped the found ADE words and phrases on free text fields of the EHR-s, using the EstNLTK toolkit [25]. In total, there were 1436 items in the ADE symptom list for AD and 1208 items for AP. Appendix B. Prefiltering rules A natural language description of each rule together with an example that this rule catches is presented here. The word/phrase in bold indicates the part in text that the corresponding rule detected. Rule 1: the text snippet contains the Estonian words ‘‘kõrvalmõju’’, ‘‘kõrvaltoime’’, ‘‘kõrvalnäht’’ (all meaning ‘‘adverse event’’) in any of their morphological or slightly altered forms. (1) Pt-l Patient-ADE leukopeenia. leukopenia. Ilmselt Probably klosapiini clozapine-GEN kõrvaltoime. side.effect ‘The patient has leukopenia. Probably a side effect of clozapine.’ 3https://ravimiregister.ee/?pv=PublicDownloads Informatics in Medicine Unlocked 59 (2025) 101701 7 D. Särg et al. Rule 2: the text snippet contains the drug name in an elative case as this is how ADEs are commonly reported in Estonian. (2) Suukuivus dry.mouth amitriptylinist. amitriptyline-ELA ‘Dry mouth from amitriptyline.’ Rule 3: the text snippet contains abbreviations ‘‘kt’’, ‘‘km’’ (abbreviations of the aforementioned Estonian terms for ADEs) or the drug name is in comitative case. (3) Olanzap. olanzapine uimasus, drowsiness raskesti difficultly talutavad tolerable KT-d side.effects ravist. treatment-ELA ‘Drowsiness from olanzapine, difficult to tolerate side effects from the treatment.’ Rule 4: the text snippet contains words/phrases that indicate ending or switching treatment or the fact that the patient complains or there is a problem. (4) Sai received ravi treatment Mirtazapiniga, mirtazapine-COM kaebas complained kaalutõusu. weight.gain ‘Received treatment with mirtazapine, complained about weight gain.’ Appendix C. Manual annotation details Our data was annotated by one expert annotator. While ideally there would be at least two, we considered one to be sufficient as the task was relatively simple and objective (compared to e.g. sentiment classification): the annotator had to look at the text snippet with the drug and symptom marked between tags and choose one of the following categories: •0: the text snippet does not contain any adverse drug events •1: the text snippet does contain an adverse drug event and it is the marked symptom resulting from the marked drug •2: the text snippet does contain an adverse drug event but it is not between the marked drug and/or symptom •3: there is not enough information/it is impossible to tell based on the given context To quantitatively verify the ‘‘simplicity’’/‘‘objectivity’’ of the task, we extracted 200 random examples from the dataset and had a nonexpert annotate those, using the same guidelines that were presented to the expert. We calculated the inter-annotator score (Cohen kappa) between the annotators both according to the actual annotation and in a binary way where we grouped the categories 0 and 2 together as negatives, kept category 1 as positives, and omitted category 3. The multi-class Cohen kappa was 0.55 and the binary one 0.83. Cohen’s suggestion for interpretation of kappa values says that values between 0.41–0.6 suggest moderate agreement and 0.81–1.0 indicate almost perfect agreement [29]. [29] argues that, for medical studies the thresholds would be higher, with 0.40–0.59 to be considered as weak agreement and 0.8–0.9 as strong. Thus, our binary annotation falls in the ‘strong’ category while the multi-class one is to be considered as weak or, at most, moderate. Analysis of the annotation differences showed two main deficiencies in the annotation guidelines. First, we had not specified how to treat text snippets that mention an ADE that has already passed. Second, there was confusion of how to handle events when the same drug and/or symptom were mentioned multiple times inside the same text snippet. For antipsychotics, again, the manual annotation was done by one expert annotator and, for interannotator score evaluation, 200 random examples from the dataset were annotated by a non-expert as well. For this experiment, the multi-class Cohen kappa was 0.73 and the binary one 0.89. The increase compared to the first experiment was probably achieved thanks to the improved annotation guidelines prepared for both annotators. Appendix D. Model training details As the inter-annotator agreement score for multiclass annotation on our AD dataset was quite low, we decided to develop a binary model that would complement the filtering rules mentioned in the previous section. For initial training of the EstMedBert AD model, we employed two data augmentation strategies to emphasize the context around symptom-drug pairs in the text snippets. For additional positive examples, we added a manually corrected version of each text snippet where an ADE was described but the wrong symptom/drug was highlighted (annotation category 2). For additional negative examples, we added drug-symptom relations not annotated as positives and not in the manually corrected set. This resulted in a total dataset of 5651 entries — 974 positive and 4677 negative. To train a classifier, we split our AD dataset into training, development and test sets based on pseudonymised patient ID-s to avoid potential data leakages between the splits as the same information about a patient is often repeated through their different EHR-s. This way, we ended up with 4080 training, 462 development, 554 test and 555 final hold-out samples. Both models, EstMedBert and EstRoberta, were trained in a similar manner. For the model, we presented each text snippet as a pair of two texts where one contained the marked drug/substance name between ’<drg>’ tags and the other contained the symptom under consideration between ’<adr>’ tags, together with their respective contexts of a maximum of 100 characters on both sides. This approach preserved the order in which the phrases appeared in the original text as well as some information about the distance between the entities as the contexts could have a significant overlap. The training arguments and hyperparameters used in our experiments were as follows. The maximum number of training epochs was set to 100, although the best model was typically found after between 40 and 70 epochs on the development set. The batch size per device was set to 64 during both training and evaluation. We employed a learning rate of 1e-5 and a weight decay of 0.001. Additionally, we incorporated 500 warmup steps for the learning rate scheduler. The training process was configured to save checkpoints and evaluate the model at the end of each epoch. The best model was determined based on the F1 metric, and the seed was set to 41 to ensure reproducibility. Appendix E. Pharmacogenetic validation For the pharmacogenetic validation, we obtained EstBB digital drug dispensing data for antidepressants (ATC codes N06Axxx) and antipsychotics (ATC codes N05Axxx) from the Estonian National Health Insurance Fund to define controls. This data included drug prescription and purchase dates, which were analyzed to ensure adequate drug use for selecting controls. Controls were defined as individuals with at least three index drug dispenses. To qualify, the time between consecutive dispensings needed to be 90 days or less (no more than a three-month gap) at least once during their purchasing history. Additionally, only individuals with dispenses of two or fewer drugs of different active agents from the antidepressants or antipsychotics group were retained as controls. We analyzed the association between ADEs reported from Escitalopram (239 cases, 7526 controls), Sertraline (148 cases, 3303 controls), Quetiapine (232 cases and 4585 controls). We focused on the pharmacogenetic phenotypes of genes relevant in the metabolism of antidepressants or antipsychotics, CYP2C19 for Escitalopram and Sertraline andCYP2D6 for Quetiapine. Pharmacogenetic phenotypes in EstBB have previously been translated from genotype data using the PharmCAT [27] algorithm (extracted information on the CYP2C19 gene). For CYP2D6 the Stargazer algorithm [28] was used. The pharmacogenetic phenotype is determined by the two alleles a person carries (diplotype), and individuals are divided into genotype-predicted phenotype subgroups: normal (NM), ultrarapid (UM), intermediate (IM), Informatics in Medicine Unlocked 59 (2025) 101701 8 D. Särg et al. Fig. 3. Association of Antidepressant and Antipsychotic Side Effects with CYP2C19/CYP2D6 Pharmacogenetic Phenotypes. and poor metabolizers (PM), which indicate the enzymes activity of CYP2C19 and CYP2D6. The differences in reporting ADEs from AD/AP were compared between the subgroups, with CYP2C19/CYP2D6 NMs as the reference group. We also separately analyzed the most frequent star-alleles indicating a poor metabolizer status (CYP2C19*2 and CYP2D6*4). We excluded relatives to perform a logistic regression between ADE and pharmacogenetic phenotypes. One member per related individual pair (PLINK PI_HAT > 0.2) was removed, prioritizing the retention of cases. This was performed using an in-house script. Logistic regression was conducted using the R software (version 4.1.2), adjusting for sex birth year and ten principal components that were estimated based on genotype data to control for population structure. Fig. 3 highlights the odds ratios (dots) and 95% confidence intervals (CI, horizontal lines) for the association of ADEs reported due to Escitalopram (red), Sertraline (green) and Quetiapine (blue) depending on CYP2C19/CYP2D6 pharmacogenetic phenotypes. Significant associations are marked with an asterisk, and the corresponding p-values are annotated. The raw data underlying Fig. 3 is provided in Fig. 4. Appendix F. Error examples See Table 5. Appendix G. Supplementary data Supplementary material related to this article can be found online at https://doi.org/10.1016/j.imu.2025.101701. Data availability The EHR texts from EstBB cannot be shared publicly, as they may contain identifiable information about the participants. To access Informatics in Medicine Unlocked 59 (2025) 101701 9