Structuring electronic health records of breast cancer with Natural Language Processing

Procesamiento del Lenguaje Natural para la estructuración de historiales clínicos de cáncer de mama Structuring electronic health records of breast cancer with Natural Language Processing Trabajo de Fin de Grado Curso 2022–2023 Autor Álvaro García Barragán Director Víctor Robles Forcada Tutor UCM: José Ignacio Hidalgo Pérez Colaborador Ernestina Menasalvas Grado en Ingeniería Informática Facultad de Informática Universidad Complutense de Madrid Procesamiento del Lenguaje Natural para la estructuración de historiales clínicos de cáncer de mama Structuring electronic health records of breast cancer with Natural Language Processing Trabajo de Fin de Grado en Ingeniería Informática Autor Álvaro García Barragán Director Víctor Robles Forcada Tutor UCM: José Ignacio Hidalgo Pérez Colaborador Ernestina Menasalvas Convocatoria: Junio 2023 Grado en Ingeniería Informática Facultad de Informática Universidad Complutense de Madrid 29 de mayo de 2023 Dedicatoria A mi madre, por saber de primera mano que es el cáncer de mama y superarlo v Agradecimientos A todo el grupo de investigación del laboratorio MEDAL, comenzando por Víctor y Ernes, quienes desde el primer día me han hecho sentir como en casa, y terminando con todos los estudiantes con quienes he tenido la oportunidad de trabajar y crecer tanto profesionalmente como personalmente. vii Resumen Procesamiento del Lenguaje Natural para la estructuración de historiales clínicos de cáncer de mama Los historiales clínicos están escritos en lenguaje natural y, por tanto, es información no estructurada. El objetivo del proyecto es estructurar la información de historiales clínicos de pacientes con cáncer de mama de un hospital público de Madrid con el fin de conseguir información util para los medicos. De esta forma, se propone realizar la estructuración a través del uso de redes neuronales profundas para la clasificacion de las entidades: NER (Named Entity Recognition), en conjunto con otras técnicas de NLP. Finalmente se generará una base de datos semiestructurada en formato json con los historiales, que podrá ser procesada posteriormente con diferentes intenciones. Palabras clave Procesamiento del Lenguaje Natural (PNL), Aprendizaje Profundo, Transformers, Reconocimiento de entidades nombradas, Extracción de información, Extracción del diagnóstico del cáncer, Cáncer de mama ix List of figures 2.1. Spacy tokenization [1] . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2. Word embbedings represented in 3D [2] . . . . . . . . . . . . . . . . . 11 2.3. Representation of an RNN . . . . . . . . . . . . . . . . . . . . . . . 12 2.4. LSTM cell architecture [3] . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5. BILSTMlayerforNER.......................... 14 2.6. Large Language Models: A New Moore’s Law ? [4] . . . . . . . . . . 15 2.7. Comparing CNN (padding tokens are omitted), RNN, and self-attention architectures[5].............................. 16 2.8. BERTvs.GPT[6] ............................ 17 2.9. Extracting cancer concepts using BERT. . . . . . . . . . . . . . . . . 17 2.10.BERTfinetuning[6] ........................... 18 2.11.SpaCypipeline[1]............................. 21 2.12. Instance of spaCy syntactic tree . . . . . . . . . . . . . . . . . . . . . 22 2.13. Example of IOB format . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.14. Example of CONLL format . . . . . . . . . . . . . . . . . . . . . . . 26 2.15. 5-fold cross validation . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.1. Transforming Clinical Notes into Structured JSON . . . . . . . . . . 33 4.2. Corpuscollection ............................. 34 4.3. Modelscreation.............................. 34 4.4. NCP: NLP Cancer Pipeline . . . . . . . . . . . . . . . . . . . . . . . 35 4.5. Technology pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.6. Automated pipeline in batches . . . . . . . . . . . . . . . . . . . . . . 37 4.7. Pre-processing of the notes . . . . . . . . . . . . . . . . . . . . . . . . 38 4.8. Example of an annotated clinical note . . . . . . . . . . . . . . . . . . 39 4.9. Negation and Uncertainty Annotations . . . . . . . . . . . . . . . . . 41 4.10. JSON format for structuring breast cancer information . . . . . . . . 42 xvii 5.1. Graphical representation of Neg Uncert Corpus counting . . . . . . . 45 5.2. Graphical representation of Clinical corpus counting . . . . . . . . . . 46 List of tables 2.1. Number of notes per service and type . . . . . . . . . . . . . . . . . . 8 2.2. Pretrained models details [7] . . . . . . . . . . . . . . . . . . . . . . . 19 2.3. spaCyNLPPipeline ........................... 22 2.4. spaCy Command Line Interface . . . . . . . . . . . . . . . . . . . . . 23 4.1. Definition of the entities . . . . . . . . . . . . . . . . . . . . . . . . . 40 4.2. Normalization............................... 42 5.1. Model hyper-parameters . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.2. Information about NER corpus . . . . . . . . . . . . . . . . . . . . . 44 5.3. Results from Negation and Uncertainly Corpus per entity type . . . . 47 5.4. General results from Clinical Corpus . . . . . . . . . . . . . . . . . . 47 5.5. Results from Clinical corpus per entity type . . . . . . . . . . . . . . 47 5.6. Discussion of Clinical NER model . . . . . . . . . . . . . . . . . . . . 49 xix Chapter 1 Introduction “Science is not only a discipline of reason, but also of romance and passion” — Stephen Hawking Cancer remains one of the main public health problems, ranked as the leading cause of death globally [8]. According to the World Health Organization (WHO)1, cancer caused nearly 10 million deaths worldwide in 2020. In particular, breast cancer is currently the most common cancer globally, accounting for 12.5% of all new annual cancer cases worldwide2. In 2020, there were 2.3 million women diagnosed with breast cancer and 685,000 deaths around the world. The process of diagnosing and treating cancer patients generates a huge amount of information that describes symptoms, the cancer diagnosis, family history, treatments, and the evolution of the patient at the time. Physicians register this information in Electronic Health Records (EHR) using clinical notes written in narrative form. Extracting and mining this information is crucial to support oncology research, design treatment plans, and improve patient outcomes [9]. However, extracting information from clinical narratives is a challenge due to the complexity of natural language [10]. Moreover, clinical texts are written by highly skilled physicians and nurses using domain-specific terms, under time pressure, with rich and complex jargon, which makes these texts differ from those of other domains. In recent years the use of Natural Language Processing (NLP) in the biomedical domain has increased the possibility of automatically extracting information from clinical narratives. NLP is a sub-field of Computer Science (CS), and Artificial Intelligence (AI) that enables computers to understand the meaning of human natural language [11]. The application of NLP and AI techniques for processing medical records plays an increasingly significant role in advancing clinical decision support. The use of EHR to perform studies in the cancer field has also increased in the last few years. 1https://www.who.int/news-room/fact-sheets/detail/cancer 2https://www.breastcancer.org/facts-statistics 1 2Chapter 1. Introduction Currently, we are experiencing a significant boom in the field of AI, with new and revolutionary systems emerging on a regular basis. One area that is particularly growing is NLP, thanks to the release of the research paper "Attention is all you need" [12], which introduced the innovative Transformers architecture. This has led to the creation of popular language models such as ChatGPT [13], which are causing a buzz in the industry. That’s why this architecture has been chosen for the task described in this project. This project is hoped to bring some improvements to the efficiency and quality of healthcare, as well as making some progress in medical research and public health policy planning related to breast cancer. 1.1. Motivation Medical records have existed for a long time, but the formalization of electronic health records began in the early 20th century. Prior to this, doctors recorded notes in paper books and physical files, but these records were often incomplete, disorganized, and difficult to access. The formalization of medical records began in 1928 when the American Association of Medical Record Librarians and Health Information Management (AHIMA)3 established a committee to develop a uniform system of medical records. This committee developed the first edition of the International List of Medical Terms in 1932, which was designed to be a guide for consistent and uniform medical record keeping. In the following years, medical records became increasingly important for healthcare. In the 1950s, information technology began to be used in healthcare, and the use of computerized systems for medical record keeping became more common. In the 1970s, automated medical record keeping systems were introduced, which allowed doctors to input data directly into a computer. In the 1990s, internet technology began to be used for medical record keeping. This allowed doctors to share information more easily and quickly, and also allowed patients to access their medical records online. EHRs have evolved in different generations as technology has advanced. The different generations are as follows [14]: First generation: This generation emerged in the 1970s and was based on automated medical record systems. These systems were limited in their functionality and were designed primarily to store basic patient information, such as diagnoses, treatments, and test results. Second generation: In the 1980s, EHRs second-generation were developed, which were based on computerized medical record systems and offered more functionality. These systems allowed doctors to input detailed information about patient healthcare, including records of past visits, prescribed medications, and laboratory test results. 3https://www.ahima.org/ 1.1. Motivation 3 Third generation: The third generation was developed in the 1990s and focused on interconnectivity and data exchange between different healthcare systems. These systems allowed doctors to access medical information from patients of other healthcare providers and were designed to improve healthcare coordination. Fourth generation: EHRs fourth-generation emerged in the early 2000s and were based on web technology. These systems allowed doctors to access patient medical records online and also allowed patients to access their own medical information and communicate with their doctors via the web. Fifth generation4: Currently, fifth-generation is being developed. This generation will use advanced technologies such as machine learning and artificial intelligence to improve diagnostic accuracy and clinical decision-making. In this way, an EHR specifically refers to the electronic record of a patient’s medical information. Therefore, an EHR is a computer system that allows healthcare professionals to capture, store, and access a patient’s medical information electronically. This includes the patient’s medical and health information, such as medical history, diagnoses, treatments, test results, and prescribed medications. On the other hand, Health Care Information System (HCIS) refer to clinical information systems used in hospitals to manage patient information, staff, and hospital resources. These systems typically include modules for appointment scheduling, patient admission, clinical data recording, patient tracking, pharmacy management, billing management, and human resources management. Therefore, an HCIS usually includes a module for EHR registration. The Madrid Health Service (SERMAS) has a unique and centralized Electronic Health Record system in the primary care system. However, in the specialized care system (hospitals), there are different HCIS, which are specific to each hospital center. Some examples of these are: HCIS include Healthcare Provider - Health Care Information System (HP-HCIS), Electronic Medical Record System (SELENE) and Medical and Health Information Exchange (IMDH). In HCIS, information can be classified into two categories, structured and unstructured: Structured information refers to information that is stored in data fields with defined formats, making it easier for computer systems to process and analyze. Examples of structured information include patient demographics, diagnoses, test results, prescribed medications, among others. Unstructured information refers to information that does not follow a defined format and can be more difficult to process and analyze automatically. Examples of unstructured information include medical reports, progress notes, clinical observations, medical images, among others. 4https://www2.deloitte.com/us/en/insights/industry/health-care/ehr-systems-the-future-ofelectronic-health-records.html 4Chapter 1. Introduction The starting hypothesis is that it is possible to extract useful information from unstructured data in HCIS and convert it into structured information that can be utilized by computer systems. Thus, the main motivation behind this work is the development of NLP tools to extract useful information from unstructured data in HCIS. This unstructured data may include medical reports, progress notes and clinical observations. 1.2. Objectives The main objectives of this work are: To develop and evaluate NLP tools for extracting useful information from unstructured data in HCIS in spanish. To convert the extracted information into structured data that can be used by computer systems in HCIS. Develop an automated NLP pipeline that can extract the most important information from a given clinical note. These objectives entail these implications: To improve the efficiency and quality of healthcare by making detailed and accurate patient information readily available to healthcare professionals. To enhance the ability of computer systems to analyze and generate accurate and detailed reports on the health and well-being of patients. To contribute to medical research and public health policy planning by providing accurate and detailed patient information. To carry out this task of developing NLP tools to extract useful information from unstructured data in HCIS, breast cancer data from a renowned hospital will be used. These data will include medical reports, progress notes, and clinical observations related to breast cancer. Additionally, we will work in collaboration with healthcare professionals and breast cancer specialists to ensure the relevance and usefulness of the extracted and converted structured information for use in HCIS. 1.3. Work Plan In order to fulfill the goals of the project the following tasks have been defined: Identify the specific types of unstructured data that are most relevant to the project and the types of information that need to be extracted. 1.3. Work Plan 5 Conduct a thorough literature review on NLP tools and techniques for extracting information from unstructured data in HCIS. Gather and preprocess the unstructured data from the HCIS, cleaning and standardizing it for further analysis. Develop and train NLP models to extract the desired information from the unstructured data, such as named entity recognition or relation extraction. Evaluate the performance of the NLP models using appropriate metrics and adjust the models as needed. 12 Chapter 2. Materials and Techniques unstructured text data, and to label them with their corresponding categories. This is a challenging task, as named entities can be expressed in many different forms and can appear in various contexts within the text. Furthermore, people often make spelling errors, which can further complicate this task. Techniques for NER include: regex, RNNs, LSTM and BiLSTM networks. LMs like BERT have achieved state-of-the-art performance on NER tasks, thanks to the Transformers architecture. Models are pretrained on large datasets in an unsupervised manner, and then fine-tuned on smaller, task-specific datasets. All these techniques, organized by the quality of their outcomes, are detailed in this section. 2.4.1. REGEX Regular expression, also known as regex or regexp, is a pattern made up of a sequence of characters that is used to search for or match specific patterns of text. It is a powerful tool that allows you to specify a set of rules for matching and manipulating text. Regular expressions are widely used in programming, data processing, and text editing applications to search, replace, and extract text based on specific patterns. In the field of NLP, it is common to use regex for tasks such as text cleaning, normalization or NER. These techniques are widely used to identify and extract important information from text, as well as to prepare the text for further analysis or machine learning tasks. 2.4.2. RNN and LSTM Recurrent Neural Networks (RNNs) have been widely used in NLP for NER. RNNs are a type of neural network architecture that can handle sequential data, making them well-suited for NER tasks. In NER, RNNs process text inputs one token at a time, and they use the context of the previous tokens to classify each token as a named entity or not. The key advantage of RNNs is their ability to capture dependencies between the current token and previous tokens, which can be essential for accurate NER. Figure 2.3: Representation of an RNN 2.4. NER 13 The vanishing gradient problem is a common issue that arises when training Recurrent Neural Networks (RNNs) on long sequences of data. The problem arises when gradients, which are used to update the network weights during training, become extremely small as they are backpropagated through the network from the output to the input layer. When this happens, the network may not be able to learn long-term dependencies effectively and may perform poorly on long sequences. The vanishing gradient problem is particularly prevalent in RNNs because they process input sequences one element at a time, and the gradients must flow back through the same set of weights for each element in the sequence. As a result, errors can accumulate and cause gradients to become very small or even vanish over time. Unlike traditional RNNs, which suffer from the vanishing gradient problem and have difficulty learning long-term dependencies, LSTMs are designed to capture long-term dependencies by using a more complex gating mechanism that selectively remembers or forgets information over time. Long Short-Term Memory (LSTM) was introduced in 1997 by Hochreiter and Schmidhuber [21] and have since become one of the most popular and effective architectures for sequential data processing. They consist of memory cells that store information over time, input gates that control the flow of information into the cells, output gates that control the flow of information out of the cells, and forget gates that selectively discard information from the cells. Figure 2.4: LSTM cell architecture [3] Overall, LSTMs are a powerful and flexible tool for modeling sequential data and have enabled significant advances in a wide range of applications. 2.4.3. BiLSTM Bidirectional LSTM [22] (BiLSTM) is a variant of the Long Short-Term Memory (LSTM) architecture that has the ability to process input sequences in both forward and backward directions. This allows the network to capture information from both past and future contexts, making it particularly useful for sequence labeling tasks such as NER. In contrast, standard LSTMs process input sequences in only one direction, typically from past to future. While LSTMs are also effective at capturing long-term 14 Chapter 2. Materials and Techniques Figure 2.5: BILSTM layer for NER dependencies in sequences, they may not be as effective as BiLSTMs for tasks that require bidirectional context. In practice, BiLSTMs often outperform LSTMs in tasks that require bidirectional context, particularly when dealing with longer sequences of data. However, they can also be more computationally expensive and require more memory than LSTMs, due to the need to store and process information in both forward and backward directions. 2.4.4. LMs A Language Model (LM) is a statistical model that is trained on a corpus of text data to predict the likelihood of a sequence of words occurring in a language. The model assigns a probability score to each possible sequence of words in the language, based on the frequency of occurrence of those words in the training data. In other words, an LM is a machine learning algorithm that is capable of generating text that is similar to human language by modeling the statistical patterns and structures of language. The goal of an LM is to generate text that is coherent, grammatical, and meaningful, given a sequence of input words or characters. Language Models (LMs) have a wide range of applications in natural language processing, including but not limited to the following: Text Completion: LMs can be used to predict the next word or phrase in a given text, which is useful for applications such as auto-completion in text 2.4. NER 15 editors or search engines. Machine Translation: LMs can be used to translate text from one language to another by modeling the probability distribution of words in both languages. Speech Recognition: LMs can be used to convert spoken language to text by modeling the probability distribution of words in the spoken language. Sentiment Analysis: LMs can be used to analyze the sentiment of a piece of text, such as determining whether a review is positive or negative. Question Answering (Q&A): LMs can be used to answer questions posed in natural language by generating an answer based on the input text and the context of the question. Named Entity Recognition(NER): LMs can be used to learn associations between words and specific named entities, like organizations or locations. Some examples of LMs used for NER include BERT (see subsection 2.4.6) [6] and Generative Pre-trained Transformer (GPT) [23] models. These models have achieved state-of-the-art performance on NER tasks and have been used in various natural language processing applications, such as information extraction and text classification. Figure 2.6: Large Language Models: A New Moore’s Law ? [4] 16 Chapter 2. Materials and Techniques 2.4.5. Transformers Transformers is a new architecture develop by google engineers in 2017 [12], which has become the state of the art thanks to two concepts: Positional encoding is a technique used to incorporate information about the position of the tokens in a sequence into the input representation. In contrast to RNNs, which use the order of the words implicitly through their sequential processing, transformers are not designed to process inputs sequentially. Instead, they process all the words in a sequence in parallel. To incorporate position information into the input, transformers use a fixed function that maps each position in the sequence to a unique vector representation. This vector representation is added to the embedding of each word, allowing the transformer to differentiate between words based on their position in the sequence. Self attention mechanism is used to capture the relationships between words in a sequence, allows the model to focus on different parts of the input during processing. The attention matrix determines how much attention to give to each input vector when computing the weighted sum. By paying attention to the most important parts of the input, the self-attention mechanism helps the model to better capture the relationships and dependencies between different parts of the text, which can ultimately lead to more accurate predictions. Figure 2.7: Comparing CNN (padding tokens are omitted), RNN, and self-attention architectures [5] For more information about this section, please refer to my presentation available at: Link presentation. I have created this presentation to understand the technology behind transformers. 2.4. NER 17 2.4.6. BERT Bidirectional Encoder Representations from Transformers (BERT) is based on a deep neural network composed of several layers of bidirectional transformer encoders. The encoders used are similar to those found in the original transformer architecture, but differ in their bidirectional approach. Instead of processing text from left to right or right to left, it uses information from both directions to generate contextual representations of words. On the other hand, GPT Transformer [24] uses self-attention that limits each symbol’s attention to only the context to its left. Figure 2.8: BERT vs. GPT [6] The figure depicted in Figure 2.9 illustrates the utilization of BERT for the purpose of extracting named entities from clinical records. Figure 2.9: Extracting cancer concepts using BERT. 18 Chapter 2. Materials and Techniques 2.4.7. Pretraining and Fine Tuning Pretraining refers to the process of training a model on a large dataset in an unsupervised manner. The main objectives are: Learning contextualized word embeddings: learn word embeddings that capture the meaning of words based on their surrounding context. Transfer learning: model can learn general linguistic features that can be useful for a wide range of tasks. Efficiency: a model can learn useful features that can reduce the amount of data needed for fine-tuning, and can also speed up training and inference times. BERT has been pre-trained in two unsupervised tasks: Masked Language Modeling (MLM): certain tokens in a sentence are randomly masked and the model is trained to predict the original words. The percentage of tokens that are masked in BERT during training is typically set to 15% of the input tokens. Next sentence prediction (NSP): The aim of NSP is to teach the model to understand the relationship between two consecutive sentences in a text. Fine-tunning is the process of adjusting a pre-trained LM to fit on a smaller labeled dataset for a specific task through a supervised training process. Due to pretraining, the data set required is small and leads to requires less learning time. Fitting to a specific task, often requires adding a layer at the end of the model. As depicted in Figure 2.10, BERT is capable of performing tasks such as SQuAD5, NER and MNLI6. Figure 2.10: BERT finetuning [6] 5Stanford Question Answering Dataset [25] 6Multi-Genre Natural Language Inference [26] 2.4. NER 19 2.4.8. BERT pretrained setups BERT models have been pretrained with diferents configurations that are available in Hugging Face Hub7. These offer a range of choices with different sizes and pre-training characteristics enabling researchers to select the most appropriate model for their specific NLP tasks and language requirements. The table provided in Table 2.2 showcases various pre-trained models with different configurations. It is important to note that the table represents only a subset of the full range of available models. These specific configurations are primarily designed for processing Spanish or English texts. Here are some considerations for selecting a BERT model: Model Size and Capacity: Models base are smaller and have fewer parameters, making them suitable when computational resources are limited or for smallerscale tasks. On the other hand, models large are larger models with more capacity, which can be beneficial for complex tasks that require capturing fine-grained language patterns. Case Sensitivity: If preserving the original casing of words is important for your task, you should choose the cased variants. These models retain the original casing in the pre-training phase and can be useful when the distinction between uppercase and lowercase is semantically significant. Language: If your NLP task involves multiple languages, you can consider using the multilingual models. These models are trained on text from multiple languages and can handle various languages simultaneously, making them suitable for multilingual applications. However, if your task specifically involves the Spanish language, the beto models are a good option. Table 2.2: Pretrained models details [7] Model Layers Hidden Heads Params Training Data bert-base-uncased 12 768 12 110M LC English text bert-large-uncased 24 1024 16 340M LC English text bert-base-cased 12 768 12 110M Cased English text bert-large-cased 24 1024 16 340M Cased English text bert-base-multilingual-uncased 12 768 12 110M LC text in top 102 languages with largest Wikipedias bert-base-multilingual-cased 12 768 12 110M Cased text in top 104 languages with largest Wikipedias beto-base-uncased 12 1024 16 110M LC Spanish text beto-base-cased 12 1024 16 110M LC Spanish text *LC : Lower-Cased 7https://huggingface.co/ 20 Chapter 2. Materials and Techniques 2.5. NLP tool: spaCy Although there are numerous NLP tools available such as NLTK8, SparkNLP9, and Stanford CoreNLP10, we opted for spaCy due to it is developed with Python, its intuitive and user-friendly interface, and its wide range of powerful features. Thus, SpaCy11 is an open-source Python library that is widely used for NLP tasks. It has been developed by Explosion AI a company that specializes in NLP technologies. It has a number of important features. Some of the most outstanding ones are: Efficiency: is a very fast and efficient tool for processing large amounts of text, because it is programmed in Cython12. Ease of use: is easy to install and use, even for those who have no experience in natural language processing. Accuracy: uses machine learning models to analyse the text, which allows it to have a high accuracy in the analysis. Customisation: allows you to customise machine learning models to suit specific needs and improve parsing accuracy. Multilingual support: supports multiple languages, making it a useful tool for analysing text in different languages. Integration: integrates easily with other natural language processing tools and frameworks as Tensorflow 13, making it a popular choice in the development of text analysis applications and services. Support: Support for multiple NLP tasks, including NER, POS tagging, dependency parsing, and text classification Furthermore, can convert word vectors from popular tools such as FastText14 and Gensim15, or load them into any pre-trained transformer model. It is also possible to perform one’s own pre-training of the language. SpaCy’s processing pipeline involves three main components: language processing, linguistic analysis, and machine learning. The tool also comes with a CLI (see subsection 2.5.2) for simplified model development. SpaCy employs an "early stopping" technique during training. In this section, these features are explained. 8https://www.nltk.org/ 9https://sparknlp.org 10https://stanfordnlp.github.io/CoreNLP/ 11https://spacy.io/ 12https://cython.org/ 13https://www.tensorflow.org/ 14https://fasttext.cc/ 15https://radimrehurek.com/gensim/ 2.5. NLP tool: spaCy 21 2.5.1. Processing pipelines The spaCy pipeline is a sequence of components that are sequentially applied to a text to perform different NLP tasks as shown in the Figure 2.11. In spaCy, the pipeline can be customized to include only the components needed for a specific NLP task. The components in the pipeline are executed in a predefined order, but it is possible to modify the order or add new components as needed. The components of the spaCy pipeline can be divided into three main categories: Language processing components: perform preprocessing tasks, such as tokenization and lemmatization, which convert text into a more manageable structure for downstream components. Linguistic analysis components: perform more complex tasks, such as parsing and named entity identification, to extract semantic and structural information from the text. Machine learning components: use machine learning models to perform NLP tasks, such as text classification or POS labeling. These models are pre-trained on large datasets to learn linguistic patterns, and then tuned for specific tasks using smaller datasets. Table 2.3 shows the main components of spacy. However, new components can be created and added according to the needs of the application. Figure 2.11: SpaCy pipeline [1] 28 Chapter 2. Materials and Techniques is used as the test set while the remaining folds are used for training. This process helps provide a more robust estimate of the model’s performance by using all the data for both training and testing. The average performance across all iterations is calculated to assess the model’s generalization ability and detect any issues such as overfitting or underfitting. Common values for k include 5 and 10. k-fold CV =1 k k X i=1 evaluate(Mi)(2.5) Figure 2.15: 5-fold cross validation 2.8. Words similarity In NLP it is not uncommon for words to contain spelling errors. However, when it comes to structured databases, it is imperative that the words are in their correct form. To address this issue, it is essential to have a structured database that includes a comprehensive vocabulary. This allows for a comparison of each word to determine its validity. By employing this approach, spelling and vocabulary errors can be identified and corrected effectively. 2.8. Words similarity 29 There are multiple ways to compare words in a computer, and it’s important to distinguish between comparing embeddings, which is used when the words are correctly spelled. The primary methods include: Levenshtein Distance [28]: used to calculate the minimum number of singlecharacter edits (insertions, deletions, or substitutions) required to transform one string into another. It was named after the Soviet mathematician Vladimir Levenshtein, who introduced it in 1965. lev(str1, str2) =                    length(str2) iflength(str1) = 0, length(str1) if length(str2) = 0, lev(tail(str1),tail(str2)) if str1[0] = str2[0], 1 + min      lev(tail(str1), str2) lev(str1,tail(str2)) lev(tail(str1),tail(str2))      otherwise (2.6) where the tail of a string, denoted as tail(x), refers to the substring obtained by excluding the first character from the string x. Jaro-Winkler Distance [29]: This measure calculates the similarity between two words by taking into account character transpositions and prefix similarity. To compare embeddings, the Cosine distance [30] is commonly used. It measures the similarity between two numerical vectors. cosine_similarity(A,B) = A·B ∥A∥∥B∥(2.7) In order to learn how to calculate the similarity between two strings in Python, you can refer to my GitHub repository available at: https://github.com/Alvaro8gb/ Strings-similarity Chapter 3 State of the Art In recent years the use of NLP in the biomedical domain has increased the possibility of automatically extracting information from clinical narratives [31; 32; 33]. The first challenge to be addressed when extracting information from clinical texts is the identification of medical-named entities. Extracting named entities is one of the most important tasks in the medical domain since performing clinical studies commonly requires detailed patient information recorded in clinical notes [34; 10]. Clinical Named Entity Recognition (Clinical NER) is the task that aims to identify medical concepts from clinical text [32; 35]. Recently, deep learning-based approaches have shown important advances and improvements in extracting information in the biomedical domain [36; 37; 38; 39]. However, most of these proposals have focused on the English language [36; 40]. In fact, information extraction in the medical domain represents its own challenges in languages other than English [41; 42]. The main advantage of deep learning approaches is the ability to automatically learn high-level features from texts, reducing the time in the hand-crafted feature engineering process. The use of deep-learning methods has also encouraged the extraction of more detailed information related to cancer. For instance, in [43], the authors described a deep-learning approach to extract breast cancer concepts using BERT. The goal of this proposal is to extract a comprehensive set of breast cancer concepts from clinical notes written in Chinese. The authors demonstrate that the BERT-based model performs better than traditional machine learning algorithms at extracting named entities in the cancer field. In [44], the authors describe a BiLSTM-based model for clinical concept extraction from oncological clinical notes written in German. This model supports extracting several concepts such as diagnosis, treatments, and medications. Although deep learning-based approaches have improved the ability to extract medical concepts in the cancer medical field, most of these proposals have focused on the English language [36; 40] and most recently, on Chinese [43; 45]. In the Spanish language case, in [46], the authors propose Cantemist, an annotated corpus to support tumor morphology extraction. Several studies [47; 48; 49] have used this corpus to perform morphology extraction. However, the main limitation of these proposals is they only support identifying one entity type (tumor 31 32 Chapter 3. State of the Art morphology). Cancer is a complex and specialized medical field requiring a comprehensive set of medical concepts for understanding its evolution from clinical narratives [43]. In conclusion, extracting named entities from oncology clinical texts written in Spanish has not been explored deeply yet. There is a lack of corpora to support information extraction in the breast cancer domain in this language. Chapter 4 NLP Cancer Pipeline 4.1. Problem statement In Health Care Information System, data is primarily classified as structured or unstructured. Structured data, with its defined formats, is readily processable by computer systems. It encompasses patient demographics, diagnoses, test results, and prescribed medications. On the other hand, unstructured data lacks a defined format, complicating automatic processing and analysis. It typically includes medical reports, progress notes, clinical observations, and medical images. This project is based on the hypothesis that unstructured data in HCIS holds valuable information that, if extracted and transformed into structured data, could significantly benefit the health systems. The central motivation for this research is to create NLP pipeline that can effectively mine unstructured data in HCIS. The overall process is shown in Figure 4.1. Figure 4.1: Transforming Clinical Notes into Structured JSON 33 34 Chapter 4. NLP Cancer Pipeline 4.2. Structuring breast cancer information To obtain obtain a module that structure a Clinical note, we have to do other steps earlier. First of all, as show in Figure 4.2, we acquired two annotated corpus, clinical and Negation & Uncertainly. Figure 4.2: Corpus collection Subsequently, we proceeded to train the models using the corpus we had acquired earlier and evaluated their performance, as illustrated in 4.3. This specific step is detailed in chapter 5, where we demonstrate the development of two NER models, namely the Clinical and Negation & Uncertainty models. Figure 4.3: Models creation Finally, both models have been integrated into a comprehensive pipeline known as NCP, which serves as the final implementation. This module has the capability to take a clinical note in raw text format and generate a JSON file that captures the most significant concepts within the note. 4.3. NCP Stages 35 Figure 4.4: NCP: NLP Cancer Pipeline 4.3. NCP Stages In NLP, a standard workflow involves multiple steps, each dedicated to a unique function. The specifics of these stages can vary based on the problem and application at hand. Our methodology for structuring clinical notes is depicted in Figure 4.4 (showcasing the final module) and Figure 4.5 (highlighting the technologies employed). A specialized pipeline, termed NCP (NLP Cancer Pipeline) has been developed. Each clinical note traverses through every stage of the pipeline, eventually emerging in a structured format: Clinical Notes: This stage involves collecting and compiling the clinical notes from a SQL Database. Clinical notes refer to the records of patients’ medical histories, including their symptoms, diagnoses, treatments and outcomes. Preprocessing (Regex): In this stage, the clinical notes are preprocessed to prepare them for further analysis. The preprocessing step involved various steps, which are primarily accomplished through the utilization of regular expressions. The following steps are involved in this stage and described in detail in section 4.4. Corpus Annotation (Prodigy): In this stage, the clinical notes are annotated with relevant medical entities described in Appendix A. It is important to note that the annotated notes used in the annotation process are a subset of the notes described in section 2.2. This process in described in section 4.5. Transformers Networks (BERT): This stage involves the use of transformers 36 Chapter 4. NLP Cancer Pipeline networks with multilingual uncased BERT1to process the annotated corpus. This network use deep learning techniques to learn from the annotated data and generate accurate predictions. Post-processing (Words similarity): In this stage, the predictions generated by the transformers networks are post-processed to improve their accuracy and normalize the results. Primarily, he employs word similarity techniques to accomplish this task. Structured Clinical Notes (mongoDB): In this final stage, the processed clinical notes are converted into structured clinical notes. These notes are organized into a standardized format that can be easily searched and analyzed. Structured clinical notes are used for several applications such as clinical decision support, population health management, and quality improvement. Figure 4.5: Technology pipeline We developed a pipeline to efficiently process large volumes of notes by operating on batches, as shown in Figure 4.6. This approach optimizes computational efficiency and reduces overhead associated with deep learning models. At present, the pipeline is only capable of structuring notes in cases where the diagnoses within a given note are not in grammatical agreement. As an example, the clinical judgment, "presenta dos carcinomas uno ductal y otro lobulillar" would not be properly extracted by the pipeline due to the grammatical agreement between 1https://huggingface.co/bert-base-multilingual-uncased 4.3. NCP Stages 37 the terms "ductal" and "lobulillar." Although the note indicates that the patient has two cancers, the pipeline only recognizes one. In our assessment, each diagnosis represents a distinct location of the cancer. Diagnoses that identify multiple sites are therefore considered as separate diagnoses. As evidenced by the following note, the pipeline will recognize the presence of two distinct diagnoses. 2 focos: - carcinoma mucinoso infiltrante de 17 mm, g1, re negativo, rp nehativo, her2 positivo (3+), ki67 5% - carcinoma micropapilar infiltrante de 28mm, n+, g1, re 90%, rp 90%, her2 positivo (3+), ki 67 10% In order to gain a comprehensive understanding of how the pipeline functions, we have provided a detailed description in the following sections. 1 2def batch(notes, notes_index): 3 4preprocess(notes, notes_index["text"]) 5 6classificator(notes, notes_index["text"]) 7 8struct_notes =postprocess(notes, notes_index) 9 10 dump2mongo_db(struct_notes) # to JSON db 11 12 if __name__ == "__main__" : 13 14 notes, notes_index =load_db() # from SQL db 15 16 batchs =[ notes[i:i +batch_size] 17 for iin range(0,len(notes), batch_size)] 18 19 threads =[ Thread(target=batch, 20 args=(chunk, notes_index)) 21 for chunk in batchs ] 22 23 for tin threads: 24 t.start() 25 26 for tin threads: 27 t.join() Figure 4.6: Automated pipeline in batches 44 Chapter 5. Experiments and Results Table 5.1: Model hyper-parameters Parameter Value Seed 8 Accumulate gradients 3 Dropout 0.1 Optimizer Adam GPU allocator Pytorch Batch size 2000 Batch size buffer 256 Batch size discard oversize True Learn rate warmup-linear Warm up steps 250 Initial rate 0.00005 Total steps 20000 Table 5.2: Information about NER corpus Corpus Negation and Uncertainly Clinical NER Number of tokens 324116 38378 Number of entity tokens 21091 10280 Percentage label tokens 6.51% 26.79% disparity between the number of annotations and the unique words (vocabulary size) in trigger labels (NegCue, UcertCue) is more pronounced, which proves advantageous for the model’s learning process. This divergence ensures an abundant supply of samples, allowing the model to effectively comprehend the vocabulary. On the other hand, there is minimal difference observed in scope labels (NegScope, UcertScope), which is expected since negation and uncertainty can be expressed in relation to any word in natural language. This presents a challenging task for the model, as it needs to accurately identify and scope words that it hasn’t encountered in the majority of cases. In the Clinical corpus, we observe significant variation in the number of annotated words among different entities. The entities with the highest number of annotated tokens are Molecular Marker and Cancer Location, both exceeding 1500. In contrast, there are labels such as Cancer Intratype or Treatment Frequency, which have fewer than 250 annotated tokens. In addition, the Implicit Date label in the clinical corpus is particularly challenging to predict, alongside the scope labels for Negation and Uncertainty. The challenge stems from the wide range of grammatical expressions utilized to communicate temporal information, often incorporating various forms of temporal adverbs. These adverbs can be expressed in numerous ways, adding to the complexity of predicting the Implicit Date label in the clinical corpus. For example, the occurrence of an event that happened yesterday can be expressed in various ways, such as "Ayer" (Yesterday), "El día anterior" (The previous day), "Un día atrás" (One day ago), "Hace un día" (A day ago) or "En la víspera" (On the eve). 5.3. Experiments 45 Figure 5.1: Graphical representation of Neg Uncert Corpus counting 5.3. Experiments To assess the performance of the proposed approach, we employed the wellestablished standard metrics, Precision, Recall, and F-Score (refer to subsection 2.7.1). These metrics were selected due to the vital importance of Recall in NER. Incorrectly identifying an entity can result in the propagation of false information. Therefore, it is preferable to refrain from annotating an entity when in doubt, rather than annotating it incorrectly, i.e. a high Recall is better than a high Precision. To assess the effectiveness of the Clinical model, 10-fold cross validation (see subsection 2.7.2) has been selected, due to the low number of samples from some entities. The performance was calculated as the average of all ten folds executed by the cross-validation strategy. To determine the representation of the media, we calculate the standard deviation. The standard deviation is a measure of the dispersion or variability in a data set. A lower standard deviation indicates that the media is more evenly represented throughout the data set, suggesting a balanced distribution. On the other hand, a higher standard deviation implies that the occurrence of media instances is more variable, indicating potential biases or imbalances in representation. In order to evaluate the performance of the negation and uncertainty model, we employed an 80-20 split. Due to there enough representation for each entity. This means that the data set was divided into two parts: 80% for training and 20% for testing. It is important to note that splits are made in a tiered manner to ensure sufficient representation for each entity. 46 Chapter 5. Experiments and Results Figure 5.2: Graphical representation of Clinical corpus counting 5.4. Results This section presents the results of the experiments described in the previous section. The results of the Negation and Uncertainty NER model, which focuses on the identification of four entities, are shown in Table 5.3. Besides, we present the results obtained from the Clinical NER model trained on the breast cancer corpus, which includes 21 entities. Table 5.4 displays the overall results achieved by Clinical model. Following that, we will provide the performance metrics for each individual entity, as shown in Table 5.5. 5.4. Results 47 Table 5.3: Results from Negation and Uncertainly Corpus per entity type Entity Type Precision Recall F-Score Negation Concept 0.956 0.956 0.956 Negation Scope 0.836 0.823 0.829 Uncertainly Concept 0.874 0.846 0.860 Uncertainly Scope 0.748 0.708 0.728 Overall 0.873 0.859 0.866 Table 5.4: General results from Clinical Corpus Metric Mean ±Deviation Precision 0.9346 ±0.0123 Recall 0.9366 ±0.0083 F-score 0.9356 ±0.0099 Table 5.5: Results from Clinical corpus per entity type Entity Type Precision Recall F-Score Cancer Concept 0.9806 ±0.0171 0.9732 ±0.0143 0.9767 ±0.0076 Cancer Expansion 0.9801 ±0.0173 0.9796 ±0.0222 0.9798 ±0.0178 Cancer Grade 0.9325 ±0.0309 0.9373 ±0.0281 0.9347 ±0.0261 Cancer Intratype 0.963 ±0.1048 0.9798 ±0.0571 0.9667 ±0.0667 Cancer Location 0.9117 ±0.0324 0.9174 ±0.0268 0.9143 ±0.0255 Cancer Metastasis 0.9055 ±0.0284 0.9014 ±0.0375 0.903 ±0.0275 Cancer Recurrence 1.0±0.0 0.9815 ±0.0524 0.9899 ±0.0286 Cancer Stage 0.9576 ±0.0341 0.9442 ±0.0412 0.9501 ±0.0265 Cancer Subtype 0.7773 ±0.1789 0.7956 ±0.138 0.7675 ±0.1092 Cancer Type 0.9789 ±0.0133 0.9822 ±0.0144 0.9804 ±0.0092 Date 0.9596 ±0.029 0.969 ±0.0271 0.9641 ±0.0254 Implicit Date 0.7435 ±0.1084 0.7368 ±0.1123 0.7392 ±0.1064 Molec Marker 0.9343 ±0.0204 0.9506 ±0.0138 0.9423 ±0.016 Surgery 0.9296 ±0.0551 0.9328 ±0.0386 0.931 ±0.0455 TNM 0.9019 ±0.0499 0.8916 ±0.0526 0.8965 ±0.0494 Treatment 0.9321 ±0.0336 0.9235 ±0.0547 0.9268 ±0.0356 Treatment Drug 0.9289 ±0.045 0.9397 ±0.0398 0.9335 ±0.0327 Treatment Frequency 0.9841 ±0.0449 0.8889 ±0.1757 0.9221 ±0.1109 Treatment Interval 0.9348 ±0.0458 0.898 ±0.0481 0.9148 ±0.0319 Treatment Quantity 0.9008 ±0.1648 0.8581 ±0.1692 0.8673 ±0.1463 Treatment Schema 0.868 ±0.1098 0.9196 ±0.0823 0.8886 ±0.0749 48 Chapter 5. Experiments and Results 5.5. Discussion The results presented in Table 5.3 for the Negation and Uncertainty Corpus provide insights into the performance of the proposed approach for each entity type: For Negation Concept, the model achieved a precision of 0.956, recall of 0.956, and an F-score of 0.956. This indicates a high level of accuracy and completeness in identifying negation concepts. In terms of Negation Scope, the precision was measured at 0.836, recall at 0.823, and F-score at 0.829. Although slightly lower than the precision and recall for Negation Concept, the model still exhibited reasonably good performance in recognizing the scope of negation. Moving on to Uncertainty Concept, the model obtained a precision of 0.874, recall of 0.846, and an F-score of 0.860. These results suggest that there are instances where the model does not make accurate predictions. In the case of Uncertainty Scope, the precision score was 0.748, recall was 0.708, and the F-score was 0.728. The precision and recall values for this entity type are the lowest among the others, causing frequent confusion for the model. Considering the overall performance, the proposed approach falls short with an average precision of 0.873, recall of 0.859, and an F-score of 0.866. This indicates that there is room for improvement in accurately identifying negation and uncertainty concepts, as well as their respective scopes, across all entity types in the Negation and Uncertainty Corpus. The under-performance could potentially be attributed to the model being trained on different types of notes. These results suggest that the model’s performance in this area needs to be enhanced to ensure a more effective implementation of the proposed approach. The results obtained from the Clinical Corpus in Table 5.4 indicate a high level of performance for the proposed approach with precision, recall, and F-score scores consistently above 0.93. This suggests that the model is capable of accurately identifying entities while capturing a substantial portion of the relevant entities present in the data. The results obtained from the Clinical Corpus, as displayed in Table 5.5, provide valuable insights into the performance of the proposed approach for each entity type: For the Cancer Concept entity type, the model achieved a high precision of 0.9806, recall of 0.9732, and an F-score of 0.9767. These results suggest that the model demonstrates excellent accuracy and completeness in identifying cancer concepts within the clinical data. Similarly, for the Cancer Expansion entity type, the model achieved a precision of 0.9801, recall of 0.9796, and an F-score of 0.9798. These scores indicate a high level of performance in recognizing expanded cancer terminologies. 5.5. Discussion 49 Table 5.6: Discussion of Clinical NER model Entity F-score IV Vocabulary size Cancer Concept 0.98 1.00 32 Cancer Expansion 0.98 1.00 25 Cancer Grade 0.93 0.99 49 Cancer Intratype 0.97 0.98 6 Cancer Location 0.91 0.98 249 Cancer Metastasis 0.90 0.99 31 Cancer Recurrence 0.99 0.98 6 Cancer Stage 0.95 0.99 35 Cancer Subtype 0.77 0.88 26 Cancer Type 0.98 1.00 24 Date 0.96 0.45 540 Implicit Date 0.74 0.33 123 Molecular Marker 0.94 0.87 620 Surgery 0.93 0.99 88 TNM 0.90 0.38 467 Treatment 0.93 0.97 58 Treatment Drug 0.93 0.95 110 Treatment Frequency 0.92 0.92 12 Treatment Interval 0.91 0.88 81 Treatment Quantity 0.87 0.59 36 Treatment Schema 0.89 0.96 19 The Cancer Grade entity type achieved a precision of 0.9325, recall of 0.9373, and an F-score of 0.9347. These results indicate satisfactory performance in identifying cancer grades within the clinical data. Regarding the Cancer Intratype, Cancer Location, Cancer Metastasis, Cancer Recurrence, Cancer Stage, Cancer Subtype, and Cancer Type entity types, the model exhibited varying levels of precision, recall, and F-score. Overall, the model achieved relatively high scores, suggesting its capability to capture and identify these cancer-related entities effectively. For entities such as Date, Implicit Date, Molec Marker, Surgery, TNM, Treatment, Treatment Drug, Treatment Frequency, Treatment Interval, Treatment Quantity, and Treatment Schema, the model demonstrated reasonable performance, as indicated by the precision, recall, and F-score scores. Examining the performance of different entities, we found that tags with low support, such as Treatment Schema, Treatment Quantity, Treatment Interval, Treatment Frequency, and Cancer Subtype, exhibited poorer results. However, tags with a limited vocabulary, such as Treatment or Cancer Recurrence, demonstrated higher accuracy, likely due to the model’s ability to easily learn them. We present the correlation between the IV (Index of Variability) and vocabulary size variables with the 50 Chapter 5. Experiments and Results F-score in Table 5.6. The results show that, in most cases, a higher IV corresponds to better classification performance. Furthermore, we noted that the Date label consistently outperformed the Implicit Date label. The Date label, being shorter and more structured, typically included specific dates such as "23/02" or "24/03/2023," while the Implicit Date label often contained phrases like "tomorrow in the morning". In practice, regular expressions are commonly employed to extract entities like Date or TNM, leveraging their shared structural patterns. In order to tackle the challenges and enhance the overall performance of the models and their functionality within the pipeline, we propose the following recommendations: Conduct additional annotation for entities that exhibit significant deviations. This will involve carefully identifying and labeling instances where the models struggle or produce inaccurate results. Annotate the Negation and Ucertanly labels using the breast cancer corpus. By leveraging a specialized dataset specific to breast cancer, we can fine-tune the model to better understand and handle negation and uncertainty related to this particular domain. This targeted retraining will lead to improved performance and accuracy when dealing with breast cancer-related information. Explore the incorporation of regular expression to improve entity extraction in structured patterns such as Date or TNM, in conjunction with deep learning models. By implementing these recommendations, we can address the challenges faced by the models, improve their performance, and enhance the overall functionality of the pipeline. To establish the order of priorities, it is advisable to follow the Amdahl’s Law [51], which emphasizes improving modules with broader global impact. In our case, the Negation and Uncertainty model should take precedence since its erroneous detection leads to the rejection of clinical terms predicted by the Clinical NER model. Chapter 6 Conclusions and Future Work This project introduces a deep learning-centric methodology for structuring the extraction of breast cancer information. Our proposal is an automated pipeline, called NCP -NLP Cancer Pipeline-, capable of processing clinical notes in Spanish and producing a structured JSON file. The results of this approach underscore the necessity of clinical NER and the detection of negation and uncertainty as the main tasks in information structuring within the clinical domain. Leveraging deep learning-based methods for the extraction of medical concepts yielded promising outcomes in processing Spanish breast cancer clinical narratives. We proposed an annotation scheme Table 4.1, which aims to extract detailed and specific concepts for cancer diagnosis and treatments. Thus, this annotation schema is aimed at extracting medical concepts and structuring them. The deep learningbased model trained with multilingual BERT has also shown promising results in detecting negation and uncertainty in Spanish clinical texts. Specifically, we used the scope of negation and uncertainty for extracting medical concepts affected by these linguistic phenomena. 6.1. Deep Learning vs. REGEX Machine learning models are considered better for NER than regex, because they can learn more complex patterns and relationships in text data, and can better adapt to different contexts and variations in the data. In contrast, regular expressions rely on predefined text patterns and are less flexible in terms of adapting to different variations in the data. However, it is important to note that regular expressions are more interpretable1than deep learning networks for this reason regex is still widely used. The problem with regular expressions is that it can be time-consuming and difficult to define patterns that accurately match all possible variations of named entities. Additionally, it may not be able to capture named entities that do not 1A model is said to be interpretable when it is easy for humans to understand and explain how it makes predictions or decisions based on the input data. 51 52 Chapter 6. Conclusions and Future Work follow a specific pattern or structure. This presents a problem that I refer to as "the regex middle point". This issue arises when the text you’re trying to extract has numerous variations, and you need a regex that might extract words you don’t want. This problem reflects the difficulty of making a regex that fits exactly what you want. It’s recommended to consider the source of the text before deciding on an extraction method. If the text is machine-generated, regular expressions (regex) might often be adequate. However, for more complex or natural language text, training a NER model based on deep learning could be more effective. For further information and guidance on when to employ regex or a deep learning approach, refer to the following guide: https://github.com/explosion/assets/blob/main/ Prodigy/Prodigy_NER_flowchart_v2_0_0_light.pdf. 6.2. Future work In future research studies, there are several main avenues for further exploration and investigation into various aspects of the problem. These include: Consider investigating alternative language models designed for processing Spanish clinical text, such as RoBERTa2or RoBERTa Clinical3. Harnessing the power of models that have a better adaptation to Spanish clinical notes could potentially enhance the accuracy and effectiveness of the results. Consider constructing our own language model. Leveraging a pre-existing Spanish model as a foundation, we can engage in further pre-training utilizing a specialized medical corpus with an emphasis on oncology-specific language, including content from scientific research journals within this domain. This model, once completed, could be released to the public, marking a significant milestone as the first Spanish language model with a specialization in medical cancer vocabulary. Enhancing the performance of models that handle negation and uncertainty is crucial in the realm of clinical narrative analysis in spanish. The current pipeline is designed to structure notes that contain a single diagnosis. To effectively comprehend and process notes with multiple diagnoses, an effective approach could be to utilize another Language Model (LM). This LM could initially preprocess the notes, conducting a primary cleanup of the data. If multiple diagnoses are detected, the LM could then segment the text into distinct paragraphs corresponding to each diagnosis. 2https://huggingface.co/BSC-LT/roberta-base-bne 3https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es Bibliography Imagination is more important than knowledge. Knowledge is limited, while imagination is not. Albert Einstein [1] B. Srinivasa-Desikan, Natural Language Processing and Computational Linguistics: A practical guide to text analysis with Python, Gensim, spaCy, and Keras. Packt Publishing Ltd, 2018. [2] C. Allen and T. Hospedales, “Analogies explained: Towards understanding word embeddings,” in International Conference on Machine Learning. PMLR, 2019, pp. 223–231. [3] C. Olah. (2015) Understanding lstm networks. [Online]. Available: https: //colah.github.io/posts/2015-08-Understanding-LSTMs/ [4] Hugging Face, “Hugging face blog: Large language model,” https://huggingface. co/blog/large-language-models, 2021. [5] A. Zhang, Z. C. Lipton, M. Li, and A. J. Smola, “Dive into deep learning,” arXiv preprint arXiv:2106.11342, 2021. [6] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, no. Mlm, pp. 4171–4186, 2019. [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805 [8] R. L. Siegel, K. D. Miller, H. E. Fuchs, and A. Jemal, “Cancer statistics, 2022,” CA: a cancer journal for clinicians, 2022. 53 60 Appendix A. Corpus Annotation Guide 2. Implicit Date This label is used to represent dates that are expressed indirectly in the notes through time expressions Ingresado a urgencias hace 5 días. Tratado con una cirugía hace 3 años. tratamiento neodyudvante hasta agosto 2015. ingresada desde 17/08/2022 al 4/09/2020. 3. Cancer Concept This label is used to represent the different ways of referring to the term cancer in clinical notes. Mainly refers to the classification according to the type of cell that becomes cancerous. Most commons examples: Carcinoma de mama derecha multifocal. Diagnosticada de adenocarcinoma de pulmón derecho. Neoplasia de mama . presenta necrosis tumoral. Others terms: fibroma, fibrotecoma, fibroadenoma, mioma, linfoma, sarcoma, leucemia. 4. Cancer Expansion carcinoma de mama derecha multifocal. carcinoma ductal infiltrante mama derecha. carcinoma de mama lobulillar in situ. Others terms: invasivo, microinfiltrante, multicéntrico, bifocal o infiltrante multifocal. 5. Cancer Type This label represents the subcategories of cancer according to their distinctive features and characteristics. Each subtype may have different growth patterns, clinical behaviour, response to treatment and prognosis. There are the most important types for doctors. carcinoma de mama lobulillar in situ. carcinoma de mama intraductal. carcinoma infiltrante tipo histologico no especifico. Others terms: lobular, ductal, apocrino, neoroendocrino, endometrioide, adenoide, inespecífico, enfermedad de paget, cistoadenoma, filoides. 6. Cancer Subtype This label represents the other subcategories of cancer. 61 carcinoma ductal convencional infiltrante mama derecha. carcinoma mucinoso de mama derecha. Others terms: clásico/convencional, medular, papilar, tubular, mucosino/coloide/mucinoso/mucoso/mucinosa, comedoniano, sólido, cribiforme, micropapilar, plano, pleomórfico/a, folicular. 7. Cancer Grade This label refers to the grade of the cancer. It may be nuclear or Nottingham grade. In addition, either the grade number or the expression. carcinoma infiltrante inespecífico grado histológico 1 de nottingham por puntuación de 5 (2+2+1). carcinoma de mama lobulillar infiltrante, g2, re 100%. con componente intraductal asociado de alto grado nuclear. 8. Cancer Intratype There are 4 molecular subtypes of breast cancer, defined by the molecular markers. luminal A luminal B HER2 sobreexpresado Triple negativo 9. Cancer Location This label expresses the organ where the cancer manifests itself. Carcinoma lobulillar infiltrante en mama izquierda. Ca de mama izquierda cuadrante superior. ca lobulillar origen mamario. carcinoma lobulillar bilateral. mastectomía de mama derecha. cáncer de pulmón. cáncer de colon. 10. Cancer Metastasis This label includes terms referring to the spread of cancer to other organs. ganglios linfaticos con metastasis. progression nodular. invasión perineural. 62 Appendix A. Corpus Annotation Guide infiltración linfovascular. 11. Cancer Recurrence This label indicates whether the patient has relapsed after eradication of the cancer. Recaída local derecha de carcinoma lobulillar en noviembre de 2016. Recidiva local de carcinoma ductal infiltrante de mama derecha. 12. Cancer Stage This label is used to represent the stage of the cancer. Most cancers have four stages: stages 0, I (1) to IV (4). Stage I represents the early stage and stage IV represents the most advanced stage of the tumour. These stages can also be combined with the letters A, B or C as shown below. Adenocarcinoma de mama estadio IV. Carcinoma escamoso estadio I-B. Cáncer de pulmón no microcítico estadio IA. Carcinoma de mama al menos estadio iii. 13. TNM This label is used to represent the stage of the cancer using the malignant tumour classification: TNM notation. This notation represents a cancer concept using three alphanumeric codes: Tumour (T) describes the size of the tumour. The letter (N) describes the lymph nodes that are affected. The letter (M) is used to represent whether metastases are present. TNMs after neoadjuvant treatment start with y. Paciente con cáncer de pulmón cT3cN3cM1. Carcinoma de mama T3 N2 M0. 14. Molec Marker This label refers to molecular markers that are measured in patients, and whose result can be positive, negative or a percentage value. Their study provides information about the cancer and the treatment to be followed. This will include: Hormone receptors: oestrogen (re or er), progesterone (rp or pr) or both hormone receptors (rh or rrhh). ki67. Shall be accompanied by a percentage. her2/Neu/ErB2/CerB2. Accompanied by the sign/pending/inconclusive. Examples: 63 carcinoma ductal infiltrante mama derecha re y rp positivos. adenocarcinoma re 90%, rp 95%, her2 -, ki-67 1%. 15. Treatment This label includes all types of treatment that can be given to the patient, without going into any detail, i.e. only generic treatment words. Inicia tratamiento con quimioterapia. Tratamiento conservador. Tratamiento con hormona. Tratamiento radioterápico. Le han propuesto quimioterapia que la enferma ha rehusado. Tiene previsto hormonoterapia tras la irradiación. tratada con radioterapia adyuvante. 16. Treatment Drug This label is used to identify specific drug names used to treat cancer patients. Paciente diagnosticado con carcinoma, tratado con cisplatino en julio de 2017. Hormonoterapia con anastrozol. En tratamiento con herceptín. Inicia ensayo clínico con taxol. tratamiento con inhibidores de la aromatasa. 17. Treatment Frequency This label refers to the frequency with which the treatment is applied. Paciente que recibe QT CON Cisplatino 75 mg/m2 cada 21 días. La dosis total administrada ha sido de 50 gy día. 18. Treatment Interval This label refers to the total interval over which a given treatment is administered. tiene platinificado radioterapia y goserelina, tamoxifeno durante 5 anos. por 4 ciclos. txol x4 semanal. 19. Treatment Quantity It is used to represent the dose of a medicine administered to a patient at a given time. 64 Appendix A. Corpus Annotation Guide Paciente que recibe qt con cisplatino 75 mg/m2 cada 21 días. Recibe RT radical de 50 Gy el 15/12/2014. Radioterapia de 60 gy. 20. Treatment Schema This label refers to regimens or schedules used in chemotherapy for the treatment of cancer. Possible examples include: EP (etoposide and cisplatin) FEC ( Fluorouracil, epirubicin hydrochloride and cyclophosphamide) AC ( Adriamycin and cyclophosphamide) ET (etoposide and carboplatin) ICE (ifosfamide, carboplatin and etoposide) CAV (cyclophosphamide, doxorubicin or adriamycin and vincristine). CMF (cyclophosphamide, methotrexate, fluorouracil). FOLFOX •FOL (folinic acid or leucovorin) •F (fluorouracil) •OX (oxaliplatin) Examples: tratada con quimioterapia cmf. 21. Surgery This label is used to label cancer-related surgical procedures. Examples: Linfadenectomía superior izquierda. Mastectomía simple en mama derecha. Realizada tumorectomía en mama derecha. Reconstrucción con dorsal ancho y protesis. cirugía conservadora con linfadenectomía. mastectomía de reconstrucción de la mama derecha. Appendix B List of medical acronyms This is the list of acronyms used in the pre-processing of clinical notes. It is important to note that these acronyms are not universal, but rather that are specific to the doctors working in the hospital. The list is the following: ca cancer rt radioterapia fec fotoquimioterapia extracorporea qt quimioterapia ht hormonoterapia cse cuadrante superior externo csi cuadrante superior interno cie cuadrante inferior externo cii cuadrante inferior interno lsi lobulo superior izquierdo lsd lobulo superior derecho dcha derecha der derecha izq izquierda ccee cuadrantes externos ccss cuadrantes superiores cis carcinoma ductal in situ 65 66 Appendix B. List of medical acronyms cdi carcinoma ductal in situ cdis carcinoma ductal in situ nos inespecifico nst inespecifico bcg biopsia del ganglio centinela bsgc biopsia selectiva del ganglio centinela bgc biopsia ganglio centinela bscg biopsia selectiva del ganglio centinela ilv infiltracion linfovascular ipn invasion perineural ihq inmunohistoquimica op operacion gc ganglio centinela tto tratamiento neoady neoadyuvante ady adyuvante ap anatomia patologica rmi recontruccion mamaria interna rx radiografia rrhh receptores hormonales dtx docetaxel pte pendiente List of acronyms EHR: Electronic Health Records, Historias Clínicas Electrónicas NLP: Natural Language Processing, Procesamiento del Lenguaje Natural NER: Named Entity Recognition, Reconocimiento de Entidades Nombradas Clinical NER: Clinical Named Entity Recognition, Reconocimiento de Entidades Clínicas POS: Parte de la oración, Part-of-Speech CLI: Command Line Interface, Interfaz de Línea de Comandos BRAT: Brat Rapid Annotation Tool, Herramienta de Anotación Rápida Brat CONLL: Conference on Computational Natural Language Learning, Conferencia sobre Aprendizaje Computacional del Lenguaje Natural WHO: World Health Organization, Organización Mundial de la Salud AHIMA: American Association of Medical Record Librarians and Health Information Management, Asociación Americana de Bibliotecarios de Registros Médicos y Gestión de Información de Salud SERMAS: Servicio Madrileño de Salud, Madrid Health Service HCIS: Health Care Information System, Sistema de información de atención médica HP-HCIS: Healthcare Provider - Health Care Information System, Proveedor de servicios sanitarios - Sistema de información sanitaria SELENE: Electronic Medical Record System, Sistema Electrónico de Expediente y Registros Médicos IMDH: IMDH, Intercambio de Información Médica y de Salud LSTM: Long Short-Term Memory, Memoria a Corto Plazo BiLSTM: Bidirectional Long Short-Term Memory, Memoria a Corto Plazo Bidireccional 67 BERT: Bidirectional Encoder Representations from Transformers, Representaciones de Codificador Bidireccional de Transformadores BETO: BERT for Spanish, BERT para el Español RoBERTa: Robustly optimized BERT approach, BERT Robustamente Optimizado GPT: Generative Pre-trained Transformer, Transformador Pre-entrenado Generativo MLM: Masked Language Modeling, Modelado lingüístico con máscara NSP: Next sentence prediction, Predicción de la siguiente oración LM: Language Model, Modelo del Lenguaje CS: Computer Science, Ciencia de los Datos AI: Artificial Intelligence, Inteligencia Artifcial CSV: Comma-Separated Values, Valores separados por comas JSON: JavaScript Object Notation, Notación de objetos de JavaScript Neural networks are a powerful tool that can help transform health care and improve the quality of life for people around the world Jensen Huang, founder and CEO of NVIDIA