Aprendizaje profundo aplicado a la bioinformática

ESCUELA T´ ECNICA SUPERIOR DE INGENIER´ IA INFORM´ ATICA Ingenier´ıa de la Salud Aprendizaje profundo aplicado a la bioinform´atica Deep learning for bioinformatics Realizado por Guillermo L´opez Garc´ıa Tutorizado por Jos´e Manual Jerez Aragon´es Cotutorizado por Francisco Javier Veredas Navarro Departamento Lenguajes y Ciencias de la Computaci´on, UNIVERSIDAD DE M´ ALAGA M´ ALAGA, JUNIO 2018 ESCUELA T´ ECNICA SUPERIOR DE INGENIER´ IA INFORM´ ATICA GRADO EN INGENIER´ IA DE LA SALUD Aprendizaje profundo aplicado a la bioinform´atica Deep learning for bioinformatics Realizado por Guillermo L´opez Garc´ıa Tutorizado por Jos´e Manual Jerez Aragon´es Cotutorizado por Francisco Javier Veredas Navarro Departamento Lenguajes y Ciencias de la Computaci´on UNIVERSIDAD DE M´ ALAGA M´ ALAGA, JUNIO 2018 Fecha defensa: El Secretario del Tribunal Resumen: Actualmente, el aprendizaje profundo (deep learning) constituye una de las tecnolog´ıas del campo de la Inteligencia Artificial (IA) que goza de mayor ´exito y popularidad. En campos como el procesamiento de im´agenes y el an´alisis de datos secuenciales, su uso se encuentra bastante extendido, formando parte del n´ucleo de sistemas de vanguardia como los veh´ıculos de conducci´on autom´atica o los sistemas de reconocimiento facial. Sin embargo, y a pesar de sus grandes capacidades representacionales y predictivas, su aplicaci´on a problemas, como el an´alisis de datos de expresi´on para su empleo en tareas de clasificaci´on de c´ancer, en los que el n´umero de variables (N) supera con creces el n´umero de muestras (M) o patrones del conjunto de datos (NM), constituye un verdadero reto todav´ıa sin resolver. Con el objetivo de resolver este problema entre el n´umero de variables y de muestras, diferentes t´enicas de aprendizaje autom´atico de reducci´on de la dimensionalidad de los datos han sido aplicadas. Aunque esta t´ecnicas consiguen reducir el n´umero de variables, el rendimiento en predicci´on de los modelos de aprendizaje autoem´atico tradicionales es moderado, ya que el n´umero reducido de muestras empleado para el entrenamiento de los m´etodos de reducci´on de la dimensionalidad no les permite extraer las caracter´ısticas adecuadas para mejorar el rendimiento en predicci´on de forma significativa. Para resolver estos problemas y mejorar la habilidad predictiva de los m´etodos cl´asicos de aprendizaje autom´atico, proponemos un enfoque basado en el aprendizaje profundo para reducir la dimensionalidad de los datos de expresi´on, que emplea aprendizaje supervisado y no supervisado para hacer uso de todas las muestras de tumores presentes en una base de datos para resolver una tarea de clasificaci´on en c´ancer concreta. Empleando la predicci´on del subtipo intr´ınseco de c´ancer de mama como ejemplo de tarea de clasificaci´on en c´ancer, los resultados obtenidos muestran que el rendimiento de los enfoques basados en aprendizaje profundo y en t´ecnicas tradicionales de aprendizaje autom´atico son muy similares a la hora de reducir la dimensionalidad de los datos de expresi´on g´enica para su empleo en tareas de clasificaci´on en c´ancer. Sin embargo, aunque algunos enfoques tradicionales parecen superar el rendimeinto del enfoque basado en aprendizaje profundo, para concluir cu´al es el enfoque m´as efectivo m´as trabajo es necesario. Por otro lado, comparando el rendimiento de los diferentes modelos de aprendizaje profundo implementados, aunque con mucha prudencia, podemos decir que cuanto mas profundo el modelo mejor rendimiento obtuvo, mostrando el poder representacional i de estos modelos para la extracci´on de una jerarqu´ıa de representaciones abstractas utiles para la resolucion de tareas de clasificaci´on. Palabras claves: aprendizaje profundo, aprendizaje autom´atico, inteligencia artificial, datos de expresi´on g´enica, reducci´on de la dimensionalidad, c´ancer de mama, calsificaci´on Abstract: Deep learning has become one of the most promising Artificial Intelligence (AI) technologies nowadays. It has been very successfully applied to areas such as computer vision or natural language processing. However, although the great representational and predictive capabilities exhibited by these models, their feasibility to be applied to problems such as gene expression data analysis for cancer classification, in which the number of input variables (N) far exceeds the number (M) of samples (NM), remains a challenge yet to be solved. In order to solve this balancing problem, several traditional machine learning dimensioanlity reduction techniques have been applied. Although these techniques scale down the input feature space, the prediction performance of the traditional machine-learning models is moderate, as the reduced number of samples used to train both the dimensionality reduction methods and the classifiers does not allow them to extract the hidden patterns in the gene expression data in a way that improves the prediction performance significantly. In order to solve these problems and improve the prediction ability of the traditional machine-learning models, we propose a deep learning approach for reducing the dimensionality of gene expression data, which uses both unsupervised and supervised learning to make the most of the entire tumor data available in a database to solve a concrete cancer classification task. Using breast cancer intrinsic subtype prediction as an example of a cancer clasification task, the obtained results showed that the performances of both deep learning and traditional machine-learning approaches are very similar when reducing the dimensionality of gene expression data for the purpose of cancer classification. However, though some traditional approaches seem to outperform the deep ii learning strategy, to conclude which is the most effective approach more work needs to be done. On the other hand, comparing the performance of the different autoencoders, although very cautiously, we could say that the deeper the model the better perfomance was obtained, showing the representational power of deep learning to extract a hierarchy of abstract representations useful for solving classification tasks. Keywords: deep learning, machine-learning, artificial intelligence, gene expression data, dimensionality reduction, breast cancer, classification iii is analyzed by the omics disciplines, such as genomics, transcriptomics, proteomics or epigenomics. The integration of all this data from a single organism can be very valuable in fields such as medicine, where this data is used to get a holistic view of the molecular state of a patient, driving the progress of what is called P4 medicine (predictive, preventive, personalized and participatory), considered by the experts to be the medicine of the future [23]. The contributions of deep learning to this domain are mainly found in genomics (a key area in P4 medicine), due to the adaptation of the convolutional models applied in computer vision to DNA sequence data. Instead of processing 2-D images with three color channels, a DNA sequence is considered as a 1-D sequence with four channels, one for each type of nucleotide (A, C, T, G) [24]. In 2015, this approach was used by Alipanahi et al to find useful motifs in DNA sequences to predict sequence specificities of DNAand RNA-binding proteins [25]. One year later, the same convolutional strategy was used by Zhou and Troyanskaya to find effective motifs for predicting the effects of non-coding variants in DNA sequences [26]. Apart from genomics, gene-expression data analysis (transcriptomics) is becoming one of the most important omics disciplines in P4 medicine, due to the advent of high-throughput sequencing technologies such as RNA-Seq [27]. In areas such as oncology, gene expression data offers a completely new way of describing the molecular state of a patient. As cancer is considered a genetic disease, a gene expression sample from a patient (which describes the genetic changes responsible for the progression of the disease, such as the over-activity or the repression of genes) contains information of paramount importance for the prevention, diagnosis and treatment of this malignant disease. For example, in breast cancer (one of the most heterogeneous cancers with many intrinsic subtypes) the information hidden in the gene expression data, when properly extracted, can be used to diagnose the concrete subtype in a precise and effective way [28]. An accurate diagnosis is extremely important for the development of a personalized treatment, as the molecular and specific therapies as well as the predicted prognosis strongly depend on the intrinsic breast cancer subtype of the patient [29]. The contributions of deep learning to gene expression analysis for cancer prediction are extremely scarce. The reason being that, although deep learning models have demon5 strated to be able to extract the hidden patterns in extremely complex data, gene expression data present some problems that make the application of deep learning models a difficult challenge yet to be solved. Up to now, in all the successful deep learning applications we have mentioned, the data had spatial or local information (text sequences, images, biological sequences, etc.); however, this is not the case with gene expression data. To make matters worse, the dimensionality of the input feature space, i.e. the number of input features (N), is extremely high (10K-60K) in gene expression datasets. However, in clinical tasks such as cancer detection, the number of available samples (M) is very low (300-1K). This enormous imbalance between the number of input features and the number of available samples (NM) makes the learning process extremely difficult, and it is known as the curse of dimensionality [30], a common problem not only for deep learning models, but for traditional machine-learning algorithms as well. In fact, in order to solve different cancer classification tasks using gene expression data, various traditional machine-learning models have been applied, such as logistic regression, decision trees, support vector machines, swallow artificial neural networks, etc. [31, 32]. But again, the main problem faced by these algorithms is the high dimensionality of the gene expression data compared to the lack of available samples (NM). To reduce the number of input features, distinct classic dimensionality reduction techniques have been used, such as feature selection and extraction methods [33]. Although these techniques scale down the input feature space, the prediction performance of the traditional machinelearning models is moderate, as the reduced number of samples used to train both the dimensionality reduction methods and the classifiers does not allow them to extract the hidden patterns in the gene expression data in a way that improves the prediction performance significantly. This is mainly due to a scalability problem, as, for example, when using traditional machine-learning methods to predict the intrinsic breast cancer subtype from gene expression data, these supervised models cannot take advantage of any other tumor data but breast cancer, using only a few hundred samples to train the models. In order to solve these problems and improve the prediction ability of the traditional machine-learning models, we propose a deep learning approach for reducing the dimensionality of gene expression data, which uses both unsupervised and supervised learning to make the most of the entire tumor data available in a database to solve a concrete cancer 6 classification task, such as predicting the intrinsic breast cancer subtype of a patient. Now, we shall proceed to review the state of the art in deep learning for cancer detection using gene expression data. 1.2.1 Deep learning for gene expression data analysis Although, as it was said before, the contributions of deep learning to cancer prediction using gene expression data are just starting to emerge and there are not yet numerous examples, several works are worth mentioning, as they clarify how unsupervised deep learning models (essentially autoencoders) can be adapted to reduce the dimensionality of gene expression data for the purpose of cancer classification. One of the most inspiring and cited ones was done in 2013 by Fakoor et al [34]. In this work, they used a combination of PCA and simple autoencoder architectures (sparse and two-layers stacked autoencoders) to perform dimensionality reduction, and a softmax output layer on top of the autoencoder architecture during the classification stage. Although the autoencoders are constrained by the features extracted by PCA, they use a very simple linear classifier and only 2K samples for training the deep models, this is the first work using deep autoencoders and gene expression data from different tumors during the feature learning step. In 2016, Danaee et al used stacked denoising autoencoders for feature extraction, and evaluated the extracted representations performing supervised breast cancer detection [35]. Even though such a deep model was trained using only 1K breast cancer samples, this was the first time stacked denoising autoencoders were applied to gene expression data. Finally, in 2018, Way and Greene employed variational autoencoders (VAEs) to extract a latent feature space using 10K gene expression samples [36]. Though they did not use the extracted features to perform any cancer detection task, they showed that the extracted features represented biological signals. In this final project, basing on these previous works, we will try to use deep autoencoders to reduce the dimensionality of the gene expression data, as well as a transfer learning approach that allows us to train the deep learning models using a whole database of tumor samples, and use the extracted features to perform cancer classification tasks such as the prediction of breast cancer intrinsic subtypes. 7 1.3 Objectives Thus, the main objective of this project is to use a deep learning approach to reduce the dimensionality of gene expression data, as well as analyzing its effectiveness when applied to cancer classification tasks. This principal objective can be divided into two more specific objectives: 1. Adapting deep learning models to gene expression data particular difficulties. In order to solve the great imbalance between the dimensionality of the data (N) and the number of samples (NM), we will try different autoencoders architectures (sparse, stacked sparse and denoising stacked) to reduce the number of input features. In addition, we will use a transfer learning approach to increase the number of samples used during the feature extraction stage. 2. Comparing the obtained results using a deep learning approach with the ones obtained using traditional machine-learning dimensionality reduction techniques. In order to compare both approaches, we will use the extracted features to solve a cancer classification task, such as the prediction of the breast cancer intrinsic subtypes. To do that, we will use three classic supervised machine-learning algorithms: Logistic regression, support vector machines and swallow artificial neural networks. 1.4 Document structure In this section, we give a brief description of the structure of the document: •Methods: This section describes each of the phases of a traditional data-mining metodology followed to carry out this project, such as data extraction, data preprocessing, dimesnionality reduction, etc. In addition, it contains a description of the algorithms used to perform all these stages. •Results: Here, the project final results are presented and discussed. Special attention will be paid to the comparison of the results obtained using the deep learning approach and the traditional machine-learning strategy to reduce the dimensionality of the data. 8 •Conclusion: Finally, the last section contains a conclusion of the work both in English and Spanish, with a final subsection dedicated to describe future works and research lines. 9 2 Methods In this project, we compare two different strategies of solving a cancer classification task using gene expression data, in our case the prediction of breast cancer intrinsic tumor subtypes. The two approaches reduce the high dimensionality of the data in a different way, one using deep learning algorithms and the other using traditional machine-learning techniques. 2.1 Deep learning strategy The first strategy uses deep unsupervised learning models (autoencoders) to perform feature extraction. Using a transfer-learning approach, the models are pre-trained on a large compendium of gene expression samples, and then fine-tuned using a small dataset for classification purpose. Hence, we distinguish two phases: feature learning and classification learning. 2.1.1 Feature learning Data extraction To pre-train the deep models that reduce the dimensionality of the gene expression data, any of these public data sources can be used: •The Cancer Genome Atlas (TCGA) platform is a collaboration between the National Cancer Institute (NCI) and the National Human Genome Research Institute (NHGRI), that has generated one of the most complete genomic studies up to now, known as PanCancer Atlas [37]. This data contains muti-dimensional omics data (DNA methylation, gene and protein expression data, etc.) of 33 different tumor types from 11K patient samples. •The UCSC Xena portal allows to access 1521 multi-omics datasets from 135 different cohorts. The largest dataset to which they provide access to is the TCGATARGET-GTEx dataset, a data integration from three different platforms into a 10 unique dataset free of computational batch effects [38]. It comprises 20K gene expression samples, from which almost the 50% of them come from cancer patients, and the other 50% from healthy (control) patients. This makes TCGA-TARGETGTEx one of the largest and most balanced gene expression datasets, something specially useful when performing cancer detection tasks such as predicting whether a sample comes from a cancer or healthy patient. •The Gene Expression Omnibus (GEO) is a public functional genomics data repository that stores and freely distributes microarray, next generation sequencing and many other high-throughput functional genomics data provided by the scientific community. Platforms such as ARCSh4[39], provide access to all the RNA-Seq gene expression data available in GEO, processing the data from different platforms uniformly. Concretely, 187,946 samples are accessible through ARCSh4with 103,083 mouse and 84,863 human. Due to performance reasons, the dataset used in this work to pre-train the models is the Pan-Cancer gene expression dataset. Although our initial intention was to use the TCGA-TARGET-GTEx dataset, it contains almost twice (20K) the number of samples of the Pan-Cancer (11K), which makes it too large considering our hardware resources (see section ). Data preparation The original Pan-Cancer dataset contains 11K samples and 60K variables (gene transcripts). However, our hardware resources cannot process so many input variables (see section ). Hence, instead of using the original dataset, we used the data from [36] (henceforth called pan-cancer dataset), accessed through https://github.com/greenelab/tybalt. This dataset contains all the 11K samples from 33 tumor types but only includes the 5K most variably expressed genes, defined by median absolute deviation (MAD). In addition, the unit of the gene expression data is log2(FPKM + 1) transformed RSEM values. Actually, rather than using the whole pan-cancer dataset for pre-training the models, we split the data into two distinct sub-datasets: one containing only the breast cancer tumor samples (BRCA pan-cancer dataset, 1K samples) and the other containing the re11 maining samples from the rest of the 32 tumor types (non-BRCA pan-cancer dataset, 10K samples). As in our transfer-learning approach the cancer classification task we want to solve is the prediction of the intrinsic breast cancer subtype, the BRCA pan-cancer data is only used during the classification learning phase, as it contains the subtypes information, whereas the non-BRCA pan-cancer dataset is used during the feature learning phase to pre-train the deep models in an unsupervised way. The rationale behind this is that we do want to use totally different data to perform the pre-training and the fine-tuning during classification learning phase. On the other hand, regarding normalization, the non-BRCA pan-cancer data is normalised using the standard centering and scaling method (zero mean and unit variance). Dimensionality reduction To reduce the high dimensionality of the gene expression data we use autoencoders, an unsupervised feature extraction method. - Theoretical model An autoencoder, in its simplest form, is a feedforward neural network with three layers: an input, a hidden and an output layer. It is an unsupervised learning method in which the main goal is, given an input, to reconstruct an output layer representation as closely as possible to the initial input layer representation. This is done by training the network using backpropagation method to minimize the reconstruction error, a function that computes the difference between the input and the output. For example, as it is shown in Figure 3, given a set of kunlabeled training samples {x(1), x(2), . . . , x(k)}, where x(i)∈ <6, an autoencoder tries to learn a function ˆx≈x[40]. The non-linear function that transforms the input into a hidden representation is called encoder, and can be expressed as h(x) = f(W x +b), where fis the hidden activation function, such as sigmoid or tanh, Wis the hidden weight matrix and bis the bias vector of the hidden layer. The matrix Wis of dimensions n×d, where nis the dimension of the input data (number of units in the input layer), and dis the dimension of the encoded representations (number of hidden units). On the other hand, the non-linear function 12 that takes the hidden representations and transforms them into the reconstructed input representations is called decoder, and can be expressed as ˆx(h) = g(W0h+b0), where gis the output activation function, W0is the output weight matrix and b0is the bias vector of the output layer. As opposed to W, the matrix W0is of dimensions d×n. Figure 3: Example of a simple autoencoder architecture, taken from [40]. Having a hidden layer with fewer units than the input and the output layers (d<n), forces the autoencoder to compress the input representation into a lower dimensional representation, which can be reconstructed to its initial representation. That is why it is used as a dimensionality reduction method. Constraining the network, such as using a small number of hidden units, has demonstrated to force the network to extract more abstract and meaningful features in the hidden representations. In addition to reduce the number of hidden units, another popular way of constraining the network is using what is called a sparsity penalty [40]. This penalty creates sparse representations, in which hidden units tend to be inactive most of the time, i.e. close to zero if the hidden activation function is the sigmoid or ReLU 13 function and close to -1 if it is the tanh activation function. Hence, the main effect of the sparse penalty is to favour the distributed hidden (encoded) representations and the units specialization, as each input pattern is encoded by the activation of a relatively small set of hidden neurons and each neuron responds (activates) to a small set of inputs. This penalty is generally implemented using L1-regularization in the hidden layer, which is added to the reconstruction error function. Hence, if the mean squared reconstruction error is used, the overall loss function minimized during the learning procedure can be expressed as: "1 m m X i ||xi−ˆxi||2#+λ n X j d X l |wjl|(1) where mis the batch size, nis the number of input and output units, dis the number of hidden units, wjl is the weight connecting the input unit jto the hidden unit land λis the L1-regularizer penalty. The first term corresponds to the input reconstruction error, whereas the second term represents the L1-regularization, which tends to decrease the magnitude of the weights, acting as a sparsity constraint. The sparsity penalty is widely used in image processing domain, where it has shown to produce very good results [41]. Another widely used approach for constraining the network is known as denoising autoencoders [42]. During training, noise is added to the input data, and the difference between the input reconstruction and the original noiseless data is minimized using backpropagation. Hence, the goal of the network is to obtain a hidden representation robust to the introduction of noise in the input layer. In order to be able to reconstruct the input correctly, the corruption of the input data forces the network to extract more abstract and meaningful features in the hidden layer. A simple denoising autoencoder architecture can be seen in Figure 4. Finally, the last approach used to force the network to extract more abstract features is stacked autoencoders. This strategy simply consists on ”stacking” several autoencoders into a deep autoencoder model, as the one shown in Figure 5, which is the result of stacking two simple autoencoders, having a deep model of one input layer, two hidden encoder layers, and two decoder layers. The hidden abstract representation is always the 14 Hyper-parameter Possible values Number of freezed layers {1,2} Learning algorithm Stochastic Gradient Descent Learning rate {0.0001,0.0005,0.001} Momentum [0.5,0.9] Dropout2 {0,0.3,0.5} DropoutPos {1,2} Dropout1 {0,0.3,0.5} Number of epochs [40,70] Batch size [50,100] Table 6: Fine-tuning hyper-parameter space of the stacked-sparse-denoising encoder. For the Logistic Regression, Support Vector Machine and swallow Artificial Neural Network classification algorithms, their hyper-parameters are their possible values are shown in tables 7, 8 and 9. Hyper-parameter Possible values Norm penalization {L1, L2} Multiclass {One-Versus-Rest, Multinomial} Table 7: Hyper-parameter space of the Logistic Regression model. Hyper-parameter Possible values Kernel {Radial Basis Function, Polynomial} C penalty {0.1,1,10,100,1000} Gamma [1 ×10−4, 1 ×10−1] Polynomial kernel degree [2,5] Table 8: Hyper-parameter space of the Support Vector Machine model. 21 Hyper-parameter Possible values Number of units in the hidden layer {20,40,60} Hidden layer activation function {sigmoid, tanh, ReLU} Learning algorithm Stochastic Gradient Descent Learning rate [0.001,2] Momentum [0.2,0.75] Maximum number of iterations {100,200} Table 9: Hyper-parameter space of the swallow Artificial Neural Network (one-hidden layer) model. 2.2 Traditional Machine-Learning approach In order to evaluate the performance of the deep learning approach, we compared this strategy with a traditional machine-learning approach for reducing the dimensionality of the gene expression data, with the purpose of performing a cancer classification task. Data preparation The same dataset used during the classification learning stage of the previous section is used here, the BRCA pan-cancer dataset (1K samples) with the 5K most variably expressed genes as input variables. Also, the same multi-class classification task is performed, the prediction of the PAM50 breast cancer intrinsic subtypes. Dimensionality reduction and classification For reducing the dimensionality of the gene expression data, we used different classical techniques: two feature selection methods, and a feature extraction technique. The feature selection methods correspond to filter methods, one using ANOVA F-values and the other using mutual information values [33, 46]. The feature extraction method is Principal Components Analysis (PCA), a widely used dimensionality reduction technique in 22 many different domains [47]. Just like when using the deep learning approach, the gene expression data was reduced to 100 features. To be consistent with what we did when using the deep learning strategy, we again used 5-fold nested CV to evaluate the performance of the models, using the average accuracy measure. This time no fine-tuning is needed, and the features extracted by the traditional dimensionality reduction methods are used by the same classification algorithms as in the previous section: LR, SVM and swallow ANN. In this way, the inner CV is again used to tune some hyper-parameters of the classifiers (see previous section for details) using Grid-Search (LR) and Randomized-Search (SVM and ANN), and the outer CV is used to train the best selected models and evaluate their performance, by first reducing the dimensionality of the data. We also standard scaled the data (zero mean and unit variance) before feeding it into the classifiers. 23 3 Results 3.1 Deep models pre-training As we stated before, the three autoencoders models were pre-trained using the non-BRCA pan-cancer dataset. For tuning several hyper-parameters of each architecture, we simply split the data into a training (90% of the samples) and a validation set (10%), and then examine the loss curves trying different values configurations. In this way, when using the sparse architecture, we selected the values showed in Table 10. Using these values, the training and validation loss curves are the ones showed in Figure 6. The training process seems to be quite stable, though some oscillations are observed. However, overfitting clearly exists, as we can see from the distance between the training and validation loss curves. Hyper-parameter Possible values L1 regularization 1 ×10−5 Learning rate 0.001 Number of epochs 40 Batch size 100 Table 10: Hyper-parameters selected values of the sparse one-hidden layer autoencoder model. For the stacked-sparse model, the selected hyper-parameters values configuration is described in Table 11. As we can see from Figure 7, the training process seems to be quite stable too, and overfitting is again observed. Finally, when using the stacked-sparse-denoising autoencoder, the selected values of the hyper-parameters are shown in Table 12. The loss curves in Figure 8 show an incredibly stable learning process, with no observable fluctuation, as well as the almost overfitting inexistence, something really difficult to obtain in such a deep model like this one (five hidden layers). This overfitting reduction in comparison with the other two models is due 24 Figure 6: Training and validation loss curves of the sparse architecture. Layer Hyper-parameter Possible values Hidden one (encoder) L1 regularization 1 ×10−5 Hidden two (encoder) L1 regularization 1 ×10−5 Whole model Learning rate 0.0005 Number of epochs 150 Batch size 200 Table 11: Hyper-parameters selected values of the deep stacked-sparse three-hidden layers autoencoder model. to the noise used in the input layer of the denosing model, implemented using dropout, one of the most effective techniques to reduce overfitting in complex deep networks [11]. 3.2 Classification results The performance of our deep learning approach in breast cancer subtype classification is summarized in Table 13, whereas the Table 14 contains the performance of the classical machine-learning approach. All the values contained in both tables represent the average classification multi-class accuraccy obtained accros the 5 iterations of our 5-fold nested Cross-Validation procedure, and the performance of the models is evaluated in terms of the average test ACC. 25 Layer Hyper-parameter Possible values Hidden one (encoder) L1 regularization 1 ×10−5 Hidden two (encoder) L1 regularization 1 ×10−5 Hidden three (encoder) L1 regularization 1 ×10−5 Whole model Noise ratio 0.15 Learning rate 0.0001 Number of epochs 200 Batch size 225 Table 12: Hyper-parameters selected values of the deep stacked-sparse-denoising fivehidden layers autoencoder. Autoencoder Classifier Test ACC Train ACC Sparse Softmax 0.867 ±0.013 0.997 ±0.003 LR 0.864 ±0.018 0.995 ±0.005 SVM 0.879 ±0.022 0.999 ±0.001 ANN 0.849 ±0.025 0.993 ±0.014 Stacked-sparse Softmax 0.887 ±0.012 1 ±0 LR 0.878 ±0.012 1 ±0 SVM 0.888 ±0.021 1 ±0 ANN 0.888 ±0.015 1 ±0 Stacked-sparse-denoising Softmax 0.903 ±0.019 1±0 LR 0.900 ±0.011 1±0 SVM 0.893 ±0.010 1±0 ANN 0.889 ±0.011 1±0 Table 13: Performance of the classification algorithms using the deep learning approach for dimensionality reduction. In addition to the four traditional ML classifiers, we also include the softmax used to perform fine-tuning. Dimensionality reduction Classifier Test ACC Train ACC Anova LR 0.897 ±0.023 0.971 ±0.006 SVM 0.894 ±0.019 0.956 ±0.028 ANN 0.899 ±0.021 0.983 ±0.014 Mutual-Information LR 0.908 ±0.018 0.972 ±0.007 SVM 0.895 ±0.021 0.987 ±0.016 ANN 0.901 ±0.010 0.981 ±0.021 PCA LR 0.902 ±0.021 0.982 ±0.011 SVM 0.906 ±0.030 0.999 ±0.001 ANN 0.905 ±0.018 0.977 ±0.014 Table 14: Performance of the classification algorithms using the traditional machinelearning approach for dimensionality reduction. 26 Figure 7: Training and validation loss curves of the stacked-sparse architecture. Figure 8: Training and validation loss curves of the stacked-sparse-denoising architecture. When comparing the different autoencoder arquitectures, the deep models outperform the single hidden layer sparse autoencoder using any of the four classifiers. Besides, the deep stacked-sparse-denoising model obtains the best results, showing that its deep architecture and the corrupted data used during training allows the model to extract the most 27 useful features for classification purpose. However, caution is needed, as the differences between the 5-fold CV average accuracies are not statistically significant enough to draw categorical an definitive conclusions. On the other hand, comparing the classification algorithms, although the best test ACC value is obtained when using softmax classifier and stacked-sparse-denoising model, SVM seems to be most consistent when combined with autoencoder models, being the classifier that obtains the best performance when using sparse and stacked-sparse architectures. Analyzing the traditional ML approach, though the maximum test ACC value is obtained when using mutual-info (and LR), PCA seems to be the most consistent method. Comparing Table 13 and Table 14, the traditional ML strategy seems to outperform our DL approach. However, the deep stacked-sparse-denoising arquitecture outperforms Anova using any classifier. Hence, from the obtained results, we can say that, though some traditional approaches (mutual-info and PCA) seem to outperform the deep learning strategy, to conclude which is the most effective approach more work needs to be done. One of the main reasons why DL strategy does not seem to outperform the traditional ML approach is overfitting. Comparing train and test ACC, there is a much bigger difference between those values in Table 13 than in Table 14. Although dropout technique was used, other regularization methods that prevent overfitting may be needed in order to boost the performance of DL models in this particular cancer classification task. It is worth mentioning that our hardware limitations may has something to do with the fact that traditional machine-learning methods seems to outperform our deep learning approach. For the stacked-sparse and stacked-sparse-denoising deep architectures, we could only execute very few iterations of the Randomized-Search procedure to tune the hyper-parameters of the models, which could have been tuned much more effective using a greater number of iterations, possibly improving the prediction performance of those models. 28 4 Conclusion In this project, we have tried to adapt deep learning, one of the most promising and successful AI technologies nowadays, to bioinformatics domain, in particular to gene expression analaysis for cancer classification. Our main goal was to give a solution to the enormous imbalance between the number of input features (N) and the number (M) of tumor gene expression available samples (NM) using deep learning models, for the purpose of performing breast cancer intrinsic subtypes classification. In this way, we have used three distint types of autoencoders, an unsupervised feature learning technique, to reduce the dimenionality (N) of the gene expression data by a factor of 50. Besides, using a transfer-learning approach, we were able to pre-train the models using a large compendium of tumor data, different from the data used to perform the cancer classification task, and hence increasing the potential number of samples (M) used to train the models in an unsupervised way. This also allows deep learning models to extract generic features from a large tumor dataset that may be useful for solving a concrete cancer classification task such as breast cancer subtype prediction. Using a nested Cross-Validation strategy, we also compared the perfromance of the deep learning approach with a traditional machine learning strategy for reducing the dimenionality of the data. The obtained results showed that the performances of both deep learning and traditional machine-learning approaches are very similar when reducing the dimensionality of gene expression data for the purpose of breast cancer subtype classification. However, though some traditional approaches seem to outperform the deep learning strategy, to conclude which is the most effective approach more work needs to be done. On the other hand, comparing the performance of the different autoencoders, although very cautiously, we could say that the deeper the model the better perfomance was obtained, showing the representational power of deep learning to extract a hierarchy of abstract representations useful for solving classification tasks. 29 4.1 Future work In order to solve the difficulties faced by gene expression analysis classification tasks, different deep learning approaches are yet to be explored. Apart from transfer-learning, generative models such as GANs or VAEs could be used to generate artificial gene expression samples, and hence balancing the enormous disproportion of input features and available samples (NM) mentioned above. On the other hand, much more work needs to be done to exhaustively analyze the feasibility of the transfer-learning approach used in this project. We have used the extracted features by the pre-trained models to solve only one cancer classification task, but there are many more cancer prediction tasks were generic features extracted by deep learning models from a large compendium of tumor data could be very valuable. Finally, another of the most unexplored areas of deep learning is model intrepretability. Although many efforts have been made, most of deep learning models are still considered as ”black-boxes”. In areas such as bioinformatics or medicine, if we want to apply these models, interpretability must not be a lacking quality, but a characteristic. Finally, to sum up, it should be noticed that the two global objectives of the project have been greatly accomplished, and the project has served as my first experience in writing and carrying out a scientific project, in which many of the concepts learnt throughout the Bioinformatics degree have been successfully applied. 30 42. P. Vincent, H. Larochelle, Y. Bengio, P. A. Manzagol, Extracting and composing robust features with denoising autoencoders. Proceedings of the Twenty-fifth International Conference on Machine Learning, 1096–1103 (2008). 43. J. Tan, M. Ung, C. Cheng, C. S. Greene, Unsupervised feature construction and knowledge extraction from genome-wide assays of breast cancer with denoising autoencoders. Proceedings of the Pacific Symposium on Biocomputing 20, 132–143 (2015). 44. L. Gondara, Medical image denoising using convolutional denoising autoencoders. IEEE 16th International Conference on Data Mining Workshops (2016). 45. J. S. Parker et al., Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology 27, 1160–1167 (2009). 46. J. R. Vergara, P. A. Est´evez, A review of feature selection methods based on mutual information. Neural Computing and Applications 24, 175–186 (2014). 47. H. Abdi, L. J. Williams, Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics 2, 433–459 (2010). 37