Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia

Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia Por Jaime del Rey García Grado en Ingeniería Informática Facultad de Informática Directores : Óscar Garnica Alcázar y José Manuel Ruiz Giardín Análisis automático de variables categóricas de alta dimensionalidad en bases de datos médicas para la predicción de bacteriemias hospitalarias Madrid, 2020–2021 Authorization for dissemination and use The authors of this work authorize the Complutense University of Madrid to use both the code and the report produced, solely for didactic purposes and mentioning the authors of the same. Jaime del Rey García iii Acknowledgements First of all, thank Óscar and José Manuel for providing the dataset, sharing the idea and contributing to the study. A special thanks to Óscar again for his dedication and understanding throughout the process, he has made me feel that he was working side by side with me and contributed as a tutor at a level beyond academic. To all the people who have encouraged me to keep on going: to my father, my mother and my brother for their perseverance; and to my friends who have not hesitated to support me. v Sobre T EF LO NX Teflon X(cc0 1.0(documentación) MIT(código))es una plantilla de L A T EX creada por David Pacios Izquierdo con fecha de Enero de 2018. Con atribuciones de uso CC0. Esta plantilla fue desarrollada para facilitar la creación de documentación profesional para Trabajos de Fin de Grado, Trabajos de Fin de Máster o Doctorados. La versión usada es la X V:X Overleaf V2 with XeLaTeX, margin 1in, bib Contacto Autor: David Pacios Izquiero Correo: [email protected] ASCII: [email protected] Despacho 110 - Facultad de Informática vii Contents Page 1 Introduction 1 1.1 Bacteremia .................................. 1 1.2 Context .................................... 3 1.3 Objectives................................... 3 2 Work organization 5 2.1 Workplan ................................... 5 2.1.1 Development environment . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Versioncontrol ............................ 6 3 The Dataset 7 3.1 Studyofthedata............................... 7 3.1.1 Dirty categorical variables . . . . . . . . . . . . . . . . . . . . . . 12 3.1.2 Encoding String Categorical Variables . . . . . . . . . . . . . . . 14 3.2 Cleaningthedata............................... 15 3.3 Processingthedata.............................. 18 3.3.1 String similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3.2 K-NN ................................. 22 4 Categories 25 4.1 Categoryrecognition............................. 25 4.1.1 Choosing threshold value . . . . . . . . . . . . . . . . . . . . . . 25 4.1.2 Noiseremoval............................. 26 4.1.3 Identifying variants of strings . . . . . . . . . . . . . . . . . . . . 27 4.1.4 Defining the classes . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.1.5 Evaluating the results . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2 Iteratingtheprocess ............................. 34 4.3 Substituting categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3.1 Dataset insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.4 Datasetpreparation ............................. 36 5 Bias and variance 39 5.1 Avoiding Bias and Variance . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2 K-Fold Cross Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6 Random Forest 43 6.1 Fitting a Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 ix Grado en Ingeniería Informática Facultad de Informática Thus, the target of this project is to find the most relevant variables for the detection and diagnosis of bacteremia by means of ML models, including the nominal categorical variables skipped in the previous study and improving the results of the predictions in the most explainable models. Jaime del Rey García 4 Chapter 2 Work organization 2.1 Workplan Before approaching the project, a weekly follow-up was agreed through which Óscar could evaluate the progress made, giving continuity to the work. In this first contact, the development environment, the tool used for version control, as well as for the work flow, were approved. Similarly, the use of the Goole Meets tool for communication and meetings, and the use of a Google Chat room for messages between meetings is set. In the first instance, Óscar explained that it would be necessary reviewing the work carried out the previous year; reading of research on machine learning results with different coding methods applied to sparse matrices and understanding the set was given equal relevance of data with which to work and the evaluation of the variables marked as pending from the previous study. From there, the work plan was established as follows: (i) Create GitHub directory tree. (ii) Get access to the previous year repository. (iii) Reed the articles about Missing Data [8] from Yufen Ding and Jeffrey S. Simonoff, Encoding High Cardinality Variables [9] and Similarity encoding for learning with dirty categorical variables [10] from Patricio Cerda. (iv) Install Jupyter Notebook. (v) Install Python 3. (vi) Get familiar with the previous study and the script used to work with the data features. (vii) Evaluate the dataset variables. (viii) Check with the physician the categorical variables to study. (ix) Find in the resulting DataFrame object all distinct values for the study case variables. (x) Modify the script according to the needs of the current features. 5 Grado en Ingeniería Informática Facultad de Informática (xi) Create working environment workflow file with pipenv. (xii) Cleaning accents and strange characters from the study features. (xiii) Get all distinct values from the study variable Otrascomor. (xiv) Depth search for text analysis techniques and tools. (xv) Apply string similarity method to the resulting bag of words. (xvi) Identify each of the different entities from the bag of words and replace them in the DataFrame for model evaluation. (xvii) Check bag of words with 2 and 3 character minimum size abbreviations. (xviii) Create substitution list for abbreviation replacement. (xix) Run the script having applied the abbreviation substitution. (xx) Complete the workflow with the categories obtained from running the script. (xxi) Apply binary codification to all categories within the DataFrame for the study feature. (xxii) Reorganize and clean the script. (xxiii) Remove all predictor variables from the DataFrame before running Random Forest model. (xxiv) Apply Random Forest to the DataFrame with binary codification. (xxv) Run k-fold cross validation with 80/20 parameters for training and validation sets. (xxvi) Apply Random Forest to the DataFrame with binary codification. (xxvii) Retrieve graph with the weighted relevance of the features. (xxviii) Retrieve ROC graph from the Random Forest execution. (xxix) Write the project report. 2.1.1 Development environment The development environment that has been used for this project is Jupyter Notebook, with the use of Python as programming language. Óscar emphasized the requirement to maintain a file with the latest stable version of the software and the libraries used necessary for the execution of the code. The use of pipenv, a production tool that creates virtual environments and maintains the build used during development, covers this need. 2.1.2 Version control For version control, github was used which, in addition to storing the code versions, allows creating a follow-up by target cards according to the status of their completion, allowing control over all project tasks apart from development tasks. Within the project, a folder structure was established to distinguish between documentation, code, patient data and results obtained. Jaime del Rey García 6 Chapter 3 The Dataset 3.1 Study of the data This chapter will develop all the information about the dataset, the treatment it has received for its correct manipulation, as well as its interpretation and subsequent preparation for its usage with Machine Learning tools. The dataset is an anonymized dataset provided by José Manuel Ruiz, physician at the Hospital Universitario de Fuenlabrada, a 350-bed hospital with the following services: general surgery, urology, orthopaedic surgery, gynaecology and obstetrics, paediatrics, intensive care units (ICUs), haematology-oncology, internal medicine and cardiology. The database was gathered from 2005 to 2015, and it consists of 4357 anonymous patient records, a.k.a. instances, containing 117 features per patient, 49.3% female with age 65.1 ± 19.7, and 56.1% male with age 62.7 ± 20.2. Each instance contains demographic and medical data (medical history, clinical analysis, comorbidities, etc.) and the result of the blood culture, the feature to be predicted, which can take one of two values: bacteraemia and no bacteraemia. The database contains 2123 bacteraemia (51.3%), which includes aerobic, strict anaerobic and facultative anaerobic bacteria, and 2234 no bacteraemia (48.7%), including 1844 contaminations.The final classification of true bacteraemia was done in prospective time by an infectious disease physician, using all the previous data, including microbiological, clinical and analytical data. The target variables in this project are those of nominal type from the list of variables, which result in the following: desmotuci,uci,origin,otrascomor. After a first evaluation, the origin variable was discarded due to its nature. This variable represents the origin of the bacteremia, which is a feature from the final diagnosis. Features that can only be obtained after the drawning of the blood culture should not be included because in the real process the physicians would not be able to count on that values. This would also contaminate the results of the Machine Learning models. Tables 3.1 and 3.2 display the features in the dataset and the selected variables for the study. 7 Grado en Ingeniería Informática Facultad de Informática Table 3.1: Features from the dataset Features from the dataset periodo Year of the study case Ordinal mes Month of the study case Ordinal dia Day of the study case Scalar edad Age of the patient Scalar edada Age per group Ordinal edada75 Age goe to 75 Scalar edada85 Age goe to 85 Scalar Diasdet Detection time in days Scalar Prifrpos First culture to grow Scalar microrga Microorganism Scalar identif Species of the bacteremia Nominal Microrgagrupo Group of the microorganism Scalar anhonpol Anaerobes Ordinal Anaerobio Presence of anaerobes against all other bacteria Ordinal Hongos Presence of fungs against all other bacteria Ordinal Stafcoag Staphyloccocus coag Scalar Polimicr Polimicrobian Ordinal microbpoli Germs of microbial bacteria Ordinal gram Gram stain Ordinal medio True positive growth medium Ordinal frasae Growth at least in aerobes Scalar frasanae Growth at least in anaerobic flask Scalar frasextr Bottles of extracted blood cultures Scalar contamin Pollutant growth Ordinal mediocon Growth medium of the pollutant microorganism Ordinal Antibiograma Microorganism antibiogram Nominal Antiresist Categorized antibiogram Scalar glucosa Blood glucose Scalar Urea Blood urea in mg/l Scalar creatin Creatinine Scalar pcrcate Categorical PCR Scalar pcr PCR value Scalar leuc Leukocytes Scalar hgb Hemoglobin Scalar pmfn PMN percentage Scalar Jaime del Rey García 8 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Features from the dataset hbcateg Categorical Hb Scalar plaqut Platelets Scalar leucocit Leukocytosis Ordinal trombope Thrombopenia Ordinal Coagulación Altered coagulation Ordinal so Urine system analysis Scalar sedorina Urine sediment Scalar diashosp Days in hospital until bloodculture extraction Scalar lnghosp1m Hospital admissions over the last month Scalar lnghosp12m Hospital admissions of over 48h in the last year Scalar comentar Comments Nominal COMORBIL Comorbidity Scalar Diabetes Diabetes Scalar Cardiopatía Heart disease Scalar Enfresp Chronic respiratory disease Scalar Neoplasia Active neoplasia Scalar Insrenal Renal insufficiency Scalar Hepatopatia Liver disease Scalar Udvp Parenteral drug addiction Scalar Alcoholismo Alcoholism Scalar Otrascomor Other comorbidities Nominal enfbasWeinst Weinstein’s underlying disease Ordinal esteroid Steroids Ordinal drogadic Drug addiction Ordinal antibiot Antibiotics Ordinal inmunosu Immunosuppressants Ordinal neutrope Neutropenia Ordinal m_genitu Genitourinary manipulations Ordinal m_respir Respiratory manipulations Ordinal m_digest Digestive manipulations Ordinal cirugia Previous surgery Ordinal diagnost Diagnosis of bacteremia Ordinal cateter Days of last catheter placement Scalar cateter1 Type of catheter Ordinal Especialidad Specialty where the bacteremia is produced Scalar servicio Service where the bacteremia is produced Scalar 9 Grado en Ingeniería Informática Facultad de Informática Features from the dataset Ordinal urgencias Blood cultures taken in the emergency room Scalar adquisic Adquisition Ordinal durac Days of fever febore blood culture Scalar tas Systolic blood pressure Scalar tad Diastolic blood pressure Scalar fc Heart rate Scalar primtemp First ER temperature with which blood cultures are drawn Scalar temporal Temperature based on oral and axillary temperatures Scalar fiebre Fever when blood cultures are drawn Ordinal fiebrePitt Oral temperature classification in Pitt scale Scalar hipotens Hypotension Ordinal Vasopre Use of vasopressor agents at the time of bacteremia Scalar intubacion Need for intubation at the time of bacteremia Scalar RCP Cardiac resuscitation at the time of bacteremia Scalar Alerta Consciousness at the time of bacteremia Scalar PITT Pitt scale in number of ICU patients Scalar metastas Metastasis Ordinal metasta1 Where the metastasis is produed, if any Ordinal evolucio Evolution Ordinal muerte Death Ordinal origen Origin of bacteremia Nominal dxfinal Final diagnosis Scalar origensos Suspected origin of bacteremia at the time the blood cultures are drawn Scalar origenf Origin of bacteremia in the final diagnosis Scalar origenva Vascular origin Scalar atbempir Empirical antibiotic treatment Scalar t_empiri Empirical treatment adequate or inadequate Ordinal ttoesp Specific adequate treatment Scalar t_especi Specific treatment adequate or inadequate Ordinal diastto Days of treatment until start of inadequate treatment Scalar t_quirur Indication or not of surgical treatment Ordinal uci Bloodcultures drawn in ICU Nominal ucidiashem Days in ICU to blood culture drawn Scalar motuci Reason of ICU admission Scalar Jaime del Rey García 10 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Features from the dataset desmotuci Reaso of ICU admission as text Nominal consfieb Consult for fever Scalar sintomas Fever symptoms Scalar stlocal Locator syndrome Scalar dest Destination Scalar vuelta_a Return to ER Scalar tratamie Antibiotic treatment Scalar Table 3.2: Selected features in the study. Selected features desmotuci Reason for admission to ICU uci Blood cultures from ICU origin Origin of the bacteremia Otrascomor Other comorbidities A study of the variables is carried out individually to identify the characteristics of each variable. Starting with the variable otrascomor that contains the comorbidities with which each patient was admitted. Comorbidities are additional disorders that patients present in addition to the disease for which they are admitted. This variable is characteristic because it can help to identify under what conditions a patient is more likely to have bacteremia. This variable contains comorbidities as nouns separated by punctuation marks or other text characters. Comorbidities are not written in a homogeneous way, that is, some are written with diminutives, others contractions of different lengths of characters on the same word. The structure of the variable is itself a series from the pandas library where each index in the list represents the patient referred and the content within the list is a string detailing the comorbidities. 11 Grado en Ingeniería Informática Facultad de Informática [1]: v_nominales['Otrascomor'] [1]: 0deterioro cognitivo 1 2CARDIOPATIA.EPILEPSIA 3valvulopatia 4bcno, dm, hta, ci ... 5389 5390 Obesidad 5391 5392 5393 Name: Otrascomor, Length: 5394, dtype: object Figure 3.1: Sample of the feature Otrascomor. The typo of the variable is not homogeneous either, counting with upper and lower case indiscriminately. The separators, which indicate the end and the beginning of each comorbidity in succession, are also heterogeneous in a range from punctuation marks such as ’,’ or ’.’ to other types of elements such as ’+’ or ’e’. Another attribute that can be observed is the appearance of spaces as prefixes and suffixes, in addition to the indexes of the series that have no content. There are also strings throughout the series that contain accents, apostrophes and other elements that pose difficulties for the representation of the text according to what formats may be used. 3.1.1 Dirty categorical variables In the context of Machine Learning with natural language variables, dirty categories are the definition of non-curated data with high cardinality but redundancy: several categories reflect the same entity. One of the main challenges with dirty categorical variables is to identify all the elements that are related and refer to the same entity or class. Without data cleaning, different string representations of the same category will lead to completely different results or subcategories, not only because the different elements refer to the same category themselves but also because of errors such as typos that cause morphological variations. From a data-integration standpoint, these categories may be seen as a data cleaning problem about entity resolution. Tasks such as deduplication, that tries to merge different variants of the same entity, seek to recognize different variants of the same entity, which may be the best case to apply as a preprocessing step. However, data cleaning usually requires human intervention and major costs in data analysis. In this dataset almost all the examples are dirty and categorical for the variables of study. If two examples for the variable Otrascomor are taken, it can be understood that within this study there is a dirty categorical variable problem. To cater for different ways Jaime del Rey García 12 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM information might appear, several entities are shown in Figures 3.2 and 3.3 for each of the possible subclasses. [1]: anemi anemia, anemia cronica, anemica cronica ferr, anemia ferropenica, anemia hemolitica, anemia megaloblastic, anemia microcitica, anemia n-n, anemia nn, anemia normocitica normocronica, anemia por deficit b12, anemia tr cronicos Figure 3.2: Sample of the elements in the feature Otrascomor refering to anemia as dirty categories. [1]: alzheimer alzheimer avanzado, alzheimer evolucionado, alzheimer evolucionado (institucionalizada, alzheimer terminal, alzheimer., alzheimer.asma, alzheimer.dm, alzheimer.enfermedad vascular cerebral, alzheimer.itus de repeticion., ca.prostata.alzheimer avanzado.parkinson, demencia alzheimer, dm.alzheimer.tvp.itus de rep., dtalzheimer avanzado, enfermedad de alzheimer, epoc.neumonia.alzheimer., erc estadio 3. alzheimer., hemorragia ceebral; e de alzheimer, hta; alzheimer Figure 3.3: Sample of the elements in the feature Otrascomor refering to alzheimer as dirty categories. In both examples, all the variables contain the word that we might consider as the entity 13 Grado en Ingeniería Informática Facultad de Informática Figure 3.5: Example of a heat map over dirty categories of different works. The feature matrix in Figure 3.5 is computed using, at first, the bag of words with commas as separators. The presence of a large number of categories calls for representing the relationships between them so the heat map is used to that end. The results are promising after the evaluation of the first example shown in Figure 3.6, in which the map shows a high relationship value between all the chains that share the word adenocarcinoma in a heterogeneous way, such as adnocarcinoma,adenoma or adenocea. Adenocarcinoma is a term that refers to cancer. Although the different types of cancer affect the body in different ways, the result sets a clear guideline to establish adenocarcinoma as a class. Jaime del Rey García 20 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Figure 3.6: Example 1 of a heat map over 25 random dirty categories from the dataset. However, after evaluating the second example depicted in figure 3.7, a problem appears that will accompany much of the rest of the project: the map does not show a similarity value not even close to 0.4 between chains such as acv cardioembolico and acv con hemiparesia, both being a type of stroke. One relationship that is striking is the low similarity value of accidente cerebrovascular and acv. Both chains refer to the exact same concept, being able to form the same class but according to the matrix they show almost no similarity. 21 Grado en Ingeniería Informática Facultad de Informática Figure 3.7: Example 2 of a heat map over 25 random dirty categories from the dataset. 3.3.2 K-NN The K-NN algorithm is a supervised classification method that allows data subsets to be grouped by distance between the elements of each subset. This algorithm classifies each new entry in a class, according to the kneighbors closest to a certain group. To do this, it calculates the distance of the new entry to each data already existing in the model and orders those distances from lowest to highest to choose the group to which it belongs. This means that the algorithm will use the similarity between two strings to measure the distances. The group chosen will be the one that represents the shortest distance, or that group with the greatest representation in a greater longitudinal spectrum. The number of neighbors of a given example allows adjusting the noise of a classification, including in each class the points of the hyperplane that are alike the most. However, significantly reducing the number of neighbors causes an effect that may not be desired: creating classes of elements that are not necessarily different. Jaime del Rey García 22 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Figure 3.8: Example of the classes grouped relative to the number of neighbors [12] K-NN is quite sensitive to: a). The variable k, so that with different values of kwe can also obtain very different results. b). The similarity metric used, since this will strongly influence the closeness relationships that will be established in the algorithm construction process. The distance metric can contain weights that will help us calibrate the classification algorithm, making it, in effect, a custom metric. Thus, the algorithm is highly benefited by the nature of the dataset with which it is going to work. The matrix already provides a weighted metric between each pair of elements of the same. Fitting a K-NN model The next step is to fit the model to the dataset and check if there is an estimate that the algorithm can make based on the values obtained in the similarity metric. The input for the value kis chosen to be 4. The objective is to have a glance to whether the algorithm is able to group strings that the heat map did not match but that actually refer to the same concept, that is why there is no investment of resources in finding the optimal number of neighbors. from sklearn.neighbors import NearestNeighbors nn_c =NearestNeighbors(n_neighbors=4).fit(transformed_values_c) _, indices_ =nn_c.kneighbors(transformed_values_c[random_points]) indices =np.unique(indices_.squeeze()) Figure 3.9 shows of 20 selected items in the bag of words grouped. It is observed that chains such as artrosis de columna,artritis reumatoide or posible arteritis de la tem23 Grado en Ingeniería Informática Facultad de Informática poral are located nearby. This result is promising since the arthritis class is the most representative element in both cases and they have been classified as similar. On the other hand, it is observed in the same graph that elements such as poliartrosis or polinosis appear as similar elements and, although it is true that they share the prefix poli, they have widely different meanings: the first class, polyarthrosis is the definition of inflammation of several joints at the same time while polinois is an allergic disease that affects the eyes, nose and lungs produced as a reaction of the immune system to pollen. Figure 3.9: K-NN over the bag of words using k=4 neighbors Conclusion of the method The intuition behind this evaluation is that for strings with a low number of characters, the string similarity algorithm presents more noise, as the n-grams have greater weight over the total string. Jaime del Rey García 24 Chapter 4 Categories 4.1 Category recognition Despite the conclusion shown in the previous section, the elements of the bag of words are evaluated, reaching the conclusion that possibly a large number of examples have similarity values high enough to be able to work with them and obtain a reasonable number of categories that include the highest number of comorbidities exposed in the variable. In order to face the problem from a Machine Learning point of view, the next step is to be able to identify all the categories using the matrix of values and to be able to merge all the similar strings into the one that identifies the class by adapting the dataset obtained up to now to adjust it to the needs of the study. This process of adapting the dataset to the problem is manually coded as there is no tool to performs these tasks so customized as it is needed; therefore a series of functions are defined and developed that will be shown for each specific step. Each of the functions went through several tests on the dataset, so the process involved a great investment in time, this section being the bulk of the project Based on the data obtained so far, the approach that is carried out is the following: a). Set a relevance threshold for the similarity value. b). Get rid of as much noise as possible. c). Identify for each token all those that share a relevant similarity. d). Set a class for each set of strings. e). Replace each element in the original variable with its superclass. 4.1.1 Choosing threshold value The idea of establishing a threshold value for the similarity of the strings is to be able to eliminate the largest number of candidates that do not present a relationship close to the identification as an entity with each of the strings of the bag of words. 25 Grado en Ingeniería Informática Facultad de Informática This, to begin with, makes it possible to eliminate noise from the dataset by making the categories obtained represent as reliable a reflection as possible of how a professional in the field would interpret and group the dataset. The value taken by this parameter is significantly relevant for the rest of the process, with the functions and results obtained being sensitive to the increase or decrease of the threshold. Based on the first result obtained from evaluating the feature matrix from figure 3.6, a threshold value greater than 0.5 suggests that the resulting strings will necessarily be very similar to each other, ensuring a relatively small number of categories and faithful to the groups that make up each of the subsets of strings. On the other hand, looking at the results of the second heat map on figure 3.7, the choice of such a high threshold value would rule out, in this specific example, all the strings that refer to different types of acv. After assessing it with Óscar, the decision that seems the most appropriate is to establish different values for the threshold and test the execution of the entire process for each of them. Lastly, the results would be evaluated and the most convenient one would be chosen. Set of threshold values: 0.2 0.3 0.4 0.5 0.65 (4.1) 4.1.2 Noise removal The denoising process begins by using the matrix itself with the similarity values as the structure to work on. Since the goal is that all those strings that do not respect the threshold are discarded, the process carried out is similar to applying a binary encoding to the matrix or, analogously, applying a one-hot encoding to each similarity vector in which the elements that meet the condition are kept. Let 𝑥be the vector representing the first row of the matrix, with 𝑥𝑖,𝑖 ∈ [0..𝑚] being 𝑚 the total size of the bag of words. We should remember that in each row of the matrix the token of the bag of words that represents the index of the row is compared with each of the other strings. We know that, being the first token, the element 𝑥0has the value of 1 because it is compared with itself. Taking into account that the similarity values oscillate in a range of [0..1] we apply, having 𝑢as the threshold value and defining a function 𝐹on the vector such that 𝐹(𝑥𝑖)=0,∀𝑥<𝑢. This is the function shown below. Knowing that it is going to work with the elements that meet the condition, the function returns a list with the indexes of its row that have not been at 0. Jaime del Rey García 26 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM def corte_valores(lista_categorias,matriz_similitud,corte): matrix =np.empty([len(matriz_similitud),len(matriz_similitud)]) ””” Argumentos clave: lista_categorias -- objeto tipo ndarray, contiene los valores de la bolsa de palabras matriz_similitud -- objeto tipo ndarray, contiene para cada valor su valor de similitud con el resto (0..1) corte -- valor decimal (0..1) para filtrar las palabras Pone a cero todos los valores en la matriz por debajo del corte Devuelve: lista => para cada categoría las posiciones de las categorías cuya similitud es relevante (según corte) ””” for i, value in enumerate(matriz_similitud): matrix[i] =list(map((lambda x: 0if x<corte else x),matriz_similitud[i])) matriz_indices =[] for i, value in enumerate(matrix): matriz_indices.append(list(np.nonzero(matrix[i])[0])) lista_corte =[] for i, value in enumerate(matriz_indices): lista_corte.append([i,0,[j for jin matriz_indices[i]]]) return lista_corte It turns out that, as each element of the array has been compared with the rest of the elements in the same order, the values of the positions for each of the strings share the same reference, that is, if the element 𝑦has in the vector of similar tokens the index 37, this same index will represent the same string regardless of whether it appears in the vector 𝑥or 𝑧, where 𝑥,𝑦,𝑧 are entities of the word bag and, thus, have a row assigned in the matrix. This is an advantage because it avoids having to save an object for each of the strings in the word bag, saving resources in time and memory. 4.1.3 Identifying variants of strings The next step is to identify all the strings that share similarity and establish a class to group them all. Having applied the function 𝐹over the entire feature matrix, it remains in a state that can be interpreted as an adjacency matrix, giving the possibility of seeing the problem from a graph point of view. If we establish that a graph is a pair of sets 𝐺 = (𝑉,𝐴) where 𝑉is the set of vertices and 𝐴is the set of edges as pairs of the form (𝑢,𝑣) such that 𝑢,𝑣 ∈ 𝑉, we define, then, ∀𝑢,𝑣 ∈𝑉 exists a tuple of the form (𝑢,𝑣)∈𝐴 if and only if 𝑢[𝑣]>0 or 𝑣[𝑢]>0. 27 Grado en Ingeniería Informática Facultad de Informática The following illustration allows us to see how the matrix would be interpreted, if we consider that every element greater than 0 can be shown as 1. Figure 4.1: Example of graph with adjacency matrix [13] Taking the vertices and edges, we know that each edge (𝑢,𝑣) fulfills the condition that both 𝑢and 𝑣have a similarity value greater than or equal to the threshold and, therefore, they can be identified under the same category. The function corte_valores returned a list of indices in which the values were not 0 after applying the 𝐹function on a string. Considering this list 𝐴′as an adjacency list, the process to follow is to go through 𝑣,∀𝑣∈𝐴′and mark the node as visited and changing the value of the node to that of the node from which the search starts, that is, updating the index value that heads the adjacency list by the index value that started from. In the example below, an example adjacency list is considered: 0→0 1 4 6 (4.2) If we follow the algorithm stated above, then we should visit all the nodes until the next result is achieved. 0→1 7 24 146 89 (4.3) 0→4 3 (4.4) 0→6 11 43 94 159 271 (4.5) For each of the nodes in the first adjacency list, they have been visited and changed the head to 0, which is the head of the starting node. It can still be associated to which element the adjacency list belonged as the first element for each token is always the token itself. Initially, an in depth algorithm was run over each of the nodes of the graph, but string similarity is not a transitive property and it was discarded. Therefore, if a node has already been visited, its adjacency list is not traversed since it would relate elements in a transitive way and would lead to an incorrect grouping by Jaime del Rey García 28 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM classes. The following function shows the process, which returns the updated adjacency lists with the changed list head values. It is noteworthy to note that this process is only executed for those nodes that have more than one node in their adjacency list, since all will have at least one node: themselves. def aplica_categorias(posiciones_equivalencia): ””” Devuelve las categorías únicas resultantes de aplicar a la bolsa de palabras los valores de similitud con umbral ””” for i, lista_similares in enumerate(posiciones_equivalencia): posiciones_equivalencia[i][1]= 1 if len(lista_similares[2]) > 1: for pos in lista_similares[2]: if (posiciones_equivalencia[pos][1]== 0): posiciones_equivalencia[pos][0]=posiciones_equivalencia[i][0] posiciones_equivalencia[pos][1]= 1 #print(valores_cp[pos] + ' ' + str(posiciones_equivalencia[pos][1])) return np.array(posiciones_equivalencia,dtype='object') 4.1.4 Defining the classes Once the connections between all the tokens have been established, the categories that will group the rest of the strings are defined, thus dealing with the problem of dirty categories. Following the process established at the beginning of the section, it is necessary to define a map that allows identifying the category that defines or groups it for each class. With the data structures formed so far, the simplest option to develop is to establish a dictionary where the key is the index of the element to be consulted and the value is the category that groups it. This process is the one that is encoded in the following function, returning said dictionary. 29 Grado en Ingeniería Informática Facultad de Informática The third option, on the other hand, significantly increases the number of columns in the DataFrame but is much easier to understand for both models and people. Each category will form a new column in which each patient suffering from this comorbidity will have the row at 1. def aniade_categorias_por_columnas(columna_categorias, bolsa_categorias,datFrame): for i, bar1 in enumerate(bolsa_categorias): col =[0for xin range(len(datFrame))] for j, foo1 in enumerate(columna_categorias): for foo2 in foo1: if (bar1 == foo2): col[j] = 1 datFrame[bolsa_categorias[i]] =col return datFrame 4.4 Dataset preparation Finally, before carrying out the tests with the model, the choice of variables remains. In the study carried out last year, they established which variables were the most relevant for optimizing the results obtained with the models. This includes the elimination of unusable variables, as well as those that can serve as predictors for the model and, as such, must be eliminated. Table 4.1: Remaining features Features selected by the algorithm Clasifica edad sexo Polimicr anhonpol Anaerobio Hongos periodo mes dia medio microbpoli frasae frasanae frasextr COMORBIL so antibiot diashosp Especialid hgb Inghosp1m plaqut stlocal pmfn leuc Inghosp12m Neoplasia Hepatopatia Enfresp Diabetes Cardiopatia Insrenal Udvp Alcoholismo creatin sedorina enfbasWeinst drogadic inmunosu esteroid cirugia neutrope glucosa sintomas m_digest m_respir leucocit trombope m_genitu m_vascul Nonetheless there is a modification to be carried out. The feature COMORBIL is a variable that indicates whether the patient had any comorbidity at the time of hospitalization. This feature has already no use, each comorbidity will be checked for each of Jaime del Rey García 36 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM the patients as a binary variable, therefore COMORBIL ⊂Otrascomor. The feature is not needed, the information that it brings to the dataset is spread and specified over the different columns for each of the comorbidities. Leaving the variable COMORBIL in the dataset would only add redundant data and one more dimension to the dataset features, so it is removed. 37 Chapter 5 Bias and variance Supervised Machine Learning algorithms require a large volume of data that will be used during the training and testing or validation phases. The training phase in Machine Learning algorithms is where the algorithms find relationships or correlations in the data, depending on the problem and model, among the data or between the data introduced and the output expected. The testing phase consists on supplying a smaller set of data to check whether the predictions are accurate for each of the cases. It is used to measure the accuracy of the model. The procedure described above is called inductive learning. The induction capacity of a model determines the level of precision that an algorithm has when trying to solve a problem similar to those provided as an example. The goal of any Machine Learning algorithm is to deduct the training data well to any domain of the problem. The purpose of the technique is to predict future actions on never-before-seen data. The main causes of a model with unreliable accuracy are called overfitting or high variance and underfitting or high bias. As the name says, overfitting is produced when, during the learning phase, the model adjusts the weighs too tight for the training set introduced. The easiest way to recognize this situation is when a model has very good accuracy levels with the training data but surprisingly poor accuracy rates with the test data. This is usually due to the use of small datasets. On the contrary, underfitting does not achieve high accuracy rates neither with the training nor the test datasets. This can be produced due to large sets of data during the training phase and short periods of training. The algorithms then lack of time for adjusting properly the weighs and results in a generic model where no result is ”too good” nor ”too bad”. 39 Grado en Ingeniería Informática Facultad de Informática Figure 5.1: Bias and Variance [14] 5.1 Avoiding Bias and Variance Overfitting and underfitting directly affect the accuracy and reliability of the models we are working with. It is important to avoid this situations by testing the data we are going to work with [15]. • Ensure that we have a sufficient number of samples to both train the model and validate it. • Subdivide our data set and keep a portion of it to test the model. This will allow us to evaluate the performance of the algorithm and will also allow us to easily detect the effects of overfitting or underfitting. • Make sure that the test set is large enough to yield statistically meaningful results and is representative of the data set as a whole. In other words, do not pick a test set with different characteristics than the training set. • The excessive number of attributes should be avoided, since it would generate a large number of dimensions in our model. This is because each attribute makes up one dimension of the model’s sample space. The number of dimensions of the sample space must be proportional to the number of cases available to carry out the study, that is, the greater the number of attributes, the greater the number of case studies, or vice versa, in the case of having few case studies few attributes should be used. In this project, we are using the dataset left from the study carried the previous year, which has the advantages of normalized data, missing data treatment and attribute filtering. Figure 5.2 illustrates how each of the situations described above influence the decisions over a set of data and how should a model behave. Jaime del Rey García 40 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Figure 5.2: Examples of overfitting and underfitting [16] 5.2 K-Fold Cross Validation Cross-validation is a resampling procedure used to evaluate Machine Learning models on a limited data sample. It allows generating different models from the same dataset. The technique divides into different subsets from the original set and generates a model so that each subset of data is used for both the training part and the validation part. More specifically, it randomly mixes the dataframe and subdivides it into equal groups. Cross-validation is primarily used in applied Machine Learning to estimate the skill of a Machine Learning model on unseen data. That is, to use a limited sample in order to estimate how the model is expected to perform in general when used to make predictions on data not used during the training of the model. It is a popular method because it is simple to understand and because it generally results in a less biased or less optimistic estimate of the model skill than other methods, such as a simple train/test split. The general procedure is as follows: • Shuffle the dataset randomly. • Split the dataset into 𝑘groups. • For each unique group: –Take the group as a hold out or test dataset. –Take the remaining groups as a training dataset. –Fit a model on the training set and evaluate it on the test set –Retain the evaluation score and discard the model • Summarize the skill of the model using the sample of model evaluation scores 41 Grado en Ingeniería Informática Facultad de Informática Importantly, each observation in the data sample is assigned to an individual group and stays in that group for the duration of the procedure. This means that each sample is given the opportunity to be used in the hold out set 1 time and used to train the model k-1 times. K-Fold Cross validation is one of many processes used to that end, but is what will be used in this study. Figure 5.3 illustrates the process. Figure 5.3: Examples of overfitting and underfitting [17] Jaime del Rey García 42 Chapter 6 Random Forest The Random Forest algorithm is a supervised learning technique that includes different methods in the training phase. It is a model frequently used to deal with overfitting and underfitting problems. This algorithm is used for solving regression and classification problems. It has a correct operation even without adjusting its own parameters and remains stable when new data is entered. On the other hand, it requires high processing times, it is difficult to interpret and small data frames are not processed optimally. A Random Forest is an ensemble of decision trees combined with bagging. When using bagging [18], what is actually happening is that different trees see different portions of the data. The low correlation between models (trees) is the key. The reason for this effect is that the trees protect each other from their individual errors (as long as they don’t constantly all err in the same direction). No tree sees all the training data. This causes each tree to be trained with different data samples for the same problem. In this way, when combining their results, some errors are compensated for others and we have a prediction that generalizes better. When we using bagging, we also combine various Machine Learning models. Unlike other methods, the way to get errors to compensate for each other is that each model is trained with subsets of the training set. These subsets are formed by randomly choosing samples (with repetition) from the training set. 6.1 Fitting a Random Forest To adjust the model based on this classifier, it is necessary to adjust the n_estimators parameter. This parameter represents the number of trees that will make up the model and on which each case study will be evaluated. In addition, the random_state parameter is set to be able to replicate the accuracy values of the model with the same input parameters. This parameter helps to control the randomness of the algorithm when generating the decision trees. It is important while changing between datasets, parameters on the same dataset or reevaluating models. 43 Grado en Ingeniería Informática Facultad de Informática First, to avoid overfitting and underfitting, we will split the data using 80% of the data for the training process and the remaining 20% for the model validation. Taking into account the amount of samples in the dataset and the amount of attributes for each sample, the 80-20 folding distribution leaves room for a correct fitting process. from sklearn.model_selection import train_test_split X_Train, X_Test, Y_Train, Y_Test =train_test_split(X, Y, test_size=0.2) Once we have applied K-Fold Cross Validation, we can execute the code that will adjust the number of estimators to minimize the error. A range between 20 and 90 estimators is fixed. Now the model has to train using the training data from the Cross Validation method and then calculate the accuracy with the sets of data used for the test phase. arrayPred =[] for ind in range(20,90): print(ind, end =' ') RF =RandomForestClassifier(ind, random_state=0) RF.fit(X_Train, Y_Train) predicciones =RF.predict(X_Test) accuracy =accuracy_score(Y_Test, predicciones) arrayPred.append(accuracy) if accuracy >maxi_accuracy: maxi =ind maxi_accuracy =accuracy rf_max=RF In Figure 6.1 we can see that overall there are very decent accuracy values, but the peak is reached for 54 estimators (the highest value in the record is placed in position 34, as the sample goes from 20 to 90 the total number of estimators is 34+20), a random_state value of 0with an accuracy of over 0.940. For this execution, K-Fold partition with 80% for training and 20% for testing was used. The results are shown for the training phase. Jaime del Rey García 44 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Figure 6.1: Study over the number of estimators Once the returned values are known, we validate the model and the returned accuracy value in order to detect overfitting or underfitting problems. To do this, the predictions for a new dataset not used during the training phase and the accuracy of the model on this dataset are calculated. We use now the validation set. #run model with optimal parameters RF =RandomForestClassifier(54, random_state=0) RF =rf_max RF.fit(X_Train, Y_Train) #test predicciones =RF.predict(X_Test) report =pd.DataFrame() reporte_actual =classification_report(Y_Test, predicciones, output_dict=True) report['1']=reporte_actual['1'].values() Model accuracy is:0.935 In this case, slightly lower accuracy values are obtained than those previously collected, which indicates that the model does not present overfitting or underfitting problems. 6.2 Evaluation metrics In order to fully understand the degree of precision of the model in question, there are various interpretability techniques for Machine Learning models in classification problems as there can be produced different outcomes. In this example, the predicted value can 45 Grado en Ingeniería Informática Facultad de Informática had already been drawn. The features were frasae and frasanae and were also removed from the dataset. In Figure 6.11 we can see the results of the training process of the Random Forest. The model accuracy value was 0.871 after testing the model with the validation set. Figure 6.11: Study over the number of estimators with filtered features This result is far more coherent taking into account the results from the previous study that scored an accuracy of 0.86 and the list of most relevant variables, now updated in Figure 6.12, where none of the new features represent a significant decision value and therefore our model should behave almost the same. Jaime del Rey García 52 Automatic analysis of high dimensional categorical variables in medical databases for the prediction of hospital bacteremia UCM Figure 6.12: Most relevant features in the second execution 53 Chapter 7 Conclusions Working with such a heterogeneous dataset has been a real challenge for the entire dataset preparation process. The first conclusion is that the whole study and, in general, any process that is repeated similar to this, would greatly benefit from a more specific description of the text strings. Although it is a work that remains for the writing of doctors, we have seen how the substitution of, for example, stroke for cerebrovascular accident, allows the grouping and identification of categories in a much more reliable and exact way. As explained in the section on string similarity, misspellings were relevant when comparing short-length tokens, so another advantage of this situation would be the decrease in the relevance of misspellings in obtaining of the similarity between two strings. Using the file provided by Óscar has allowed the convergence of all the cleaning and data preparation work into a useful result. This file, even so, does not collect all the variants of abbreviations or spelling mistakes that the set of variables contains. This leads to the conclusion that the tool alone did not provide sufficient support to carry out the study. In short, is noticeable that natural language processing is a task that still has room for improvement, however the use of this tool has made it possible to reduce a set of more than 1000 categories to approximately one tenth. Looking at the final dataset, the amount of attributes initially added by using a binary encoding to include the categories in the dataset seemed excessive, increasing the total number of attributes to 247, since it is a perfect example for a case that may suffer high bias and generalize the weights associated with the parameters in the learning process. However, since there were more than 4,300 examples, not only has it not produced this effect, but it has also contributed to an improvement in the success rate compared to the study carried out last year. Regarding the model used, the conclusion drawn from the previous study showed a better success rate with the application of Random Forest on the dataset. The advantage of using this model over other Machine Learning models is that it is one of the models with more explanatory power, as it is directly formed by decision trees. The model resulting from this algorithm trained on the previously mentioned dataset has an accuracy of approximately 94%. One of the reasons that the fitness of the first model can be considered successful is the 55 Grado en Ingeniería Informática Facultad de Informática result of displaying the ROC curve (6.7) and the confusion matrix (6.3). The conclusion of the correct classification by the model is supported by the metrics that allow us to appreciate the clear differentiation between both distributions. We can not ignore that, after all, the process of cleaning the data, computing the similarity between all the tokens and finally obtaining the different categories from the feature Otrascomor, none of them were in the top 50 of the most weighed categories. The conclusion is that, regarding the result of the second model, with the data provided and the Machine Learning type of model used, it is rather difficult to make a correct early diagnosis of bacteremia. Jaime del Rey García 56 Chapter 8 Future improvements This project began with the idea of working on all the nominal variables of the dataset provided by Hospital Universitario de Fuenlabrada. However, the organization and work that has led to the treatment of a single variable has taken up most of this study. For this reason, one of the possible branches in which this study can lead is the one in which the rest of the variables are treated and Machine Learning methods are applied to the complemented set. Taking into account the dataset resulting from this study, it is possible that the inclusion of the other variables may overload the number of attributes for the set provided when coding them for correct treatment by Machine Learning models, yet it seems the most urgent step to take regarding the results of the last model. Other possible variants are the application of other Machine Learning models such as Neural Networks or Support Vector Machines (SVM). These models have the great disadvantage of lacking explicability. Therefore, it would require an in-depth study of the variables and the weights associated with them. However, they are models that can detect complex relationships between attributes that help improve prediction efficiency. This is specially relevant in the context of this project, where it would also be interesting to find possible relationships between the categories obtained and the original variables from the dataset. In addition, the study carried out last year and this one that complements it, provide conclusions and help to recognize bacteremia as a binary class, but in reality there are a diversity of types of bacteremia and not all share symptoms or treatment. Therefore, the conclusions drawn during these two years could be applied to the study of the detection of bacteremia as a multiclass classification problem. This particular study may be the most laborious, since it may require different datasets for each type of bacteraemia, however, if the results were successful, it would be a great boost in the prediction of bacteremia diagnoses. 57 List of Figures 1.1 Blood culture examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3.1 Sample of the feature Otrascomor. ..................... 12 3.2 Sample of the elements in the feature Otrascomor refering to anemia as dirtycategories................................. 13 3.3 Sample of the elements in the feature Otrascomor refering to alzheimer as dirtycategories................................. 13 3.4 Sample of the elements in the bag of words. . . . . . . . . . . . . . . . . 18 3.5 Example of a heat map over dirty categories of different works. . . . . . . 20 3.6 Example 1 of a heat map over 25 random dirty categories from the dataset. 21 3.7 Example 2 of a heat map over 25 random dirty categories from the dataset. 22 3.8 Example of the classes grouped relative to the number of neighbors . . . 23 3.9 K-NN over the bag of words using k=4 neighbors . . . . . . . . . . . . . 24 4.1 Example of graph with adjacency matrix . . . . . . . . . . . . . . . . . . 28 5.1 BiasandVariance............................... 40 5.2 Examples of overfitting and underfitting . . . . . . . . . . . . . . . . . . 41 5.3 Examples of overfitting and underfitting . . . . . . . . . . . . . . . . . . 42 6.1 Study over the number of estimators . . . . . . . . . . . . . . . . . . . . 45 6.2 Hypothesistesting .............................. 46 6.3 Confusionmatrix............................... 48 6.4 IdealAUCscenario.............................. 49 6.5 WorstROCscenario ............................. 49 6.6 IdealROCscenario.............................. 49 6.7 ROC curve from Random Forest . . . . . . . . . . . . . . . . . . . . . . 50 6.8 Estimated weigh per feature . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.9 Most weighed features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.10 SHAP value (impact on model output) . . . . . . . . . . . . . . . . . . . 51 6.11 Study over the number of estimators with filtered features . . . . . . . . 52 6.12 Most relevant features in the second execution . . . . . . . . . . . . . . . 53 59 List of Tables 3.1 Features from the dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Selected features in the study. . . . . . . . . . . . . . . . . . . . . . . . . 11 4.1 Remainingfeatures.............................. 36 61