ESCUELA T´
ECNICA SUPERIOR DE
INGENIER´
IA INFORM´
ATICA
Ingenie ´ıa de la Salud
Ap endizaje p o undo aplicado a la bioin o m´a ica
Deep lea ning o bioin o ma ics
Realizado po
Guille mo L´opez Ga c´ıa
Tu o izado po
Jos´e Manual Je ez A agon´es
Co u o izado po
F ancisco Ja ie Ve edas Na a o
Depa amen o
Lenguajes y Ciencias de la Compu aci´on,
UNIVERSIDAD DE M´
ALAGA
M´
ALAGA, JUNIO 2018
ESCUELA T´
ECNICA SUPERIOR DE INGENIER´
IA INFORM´
ATICA
GRADO EN INGENIER´
IA DE LA SALUD
Ap endizaje p o undo aplicado a la bioin o m´a ica
Deep lea ning o bioin o ma ics
Realizado po
Guille mo L´opez Ga c´ıa
Tu o izado po
Jos´e Manual Je ez A agon´es
Co u o izado po
F ancisco Ja ie Ve edas Na a o
Depa amen o
Lenguajes y Ciencias de la Compu aci´on
UNIVERSIDAD DE M´
ALAGA
M´
ALAGA, JUNIO 2018
Fecha de ensa:
El Sec e a io del T ibunal

Resumen:
Ac ualmen e, el ap endizaje p o undo (deep lea ning) cons i uye una de las ecnolog´ıas
del campo de la In eligencia A i icial (IA) que goza de mayo ´exi o y popula idad. En
campos como el p ocesamien o de im´agenes y el an´alisis de da os secuenciales, su uso
se encuen a bas an e ex endido, o mando pa e del n´ucleo de sis emas de angua dia
como los eh´ıculos de conducci´on au om´a ica o los sis emas de econocimien o acial. Sin
emba go, y a pesa de sus g andes capacidades ep esen acionales y p edic i as, su apli-
caci´on a p oblemas, como el an´alisis de da os de exp esi´on pa a su empleo en a eas de
clasi icaci´on de c´ance , en los que el n´ume o de a iables (N) supe a con c eces el n´ume o
de mues as (M) o pa ones del conjun o de da os (NM), cons i uye un e dade o
e o oda ´ıa sin esol e . Con el obje i o de esol e es e p oblema en e el n´ume o de
a iables y de mues as, di e en es ´enicas de ap endizaje au om´a ico de educci´on de la
dimensionalidad de los da os han sido aplicadas. Aunque es a ´ecnicas consiguen educi
el n´ume o de a iables, el endimien o en p edicci´on de los modelos de ap endizaje au-
oem´a ico adicionales es mode ado, ya que el n´ume o educido de mues as empleado
pa a el en enamien o de los m´e odos de educci´on de la dimensionalidad no les pe -
mi e ex ae las ca ac e ´ıs icas adecuadas pa a mejo a el endimien o en p edicci´on de
o ma signi ica i a. Pa a esol e es os p oblemas y mejo a la habilidad p edic i a de los
m´e odos cl´asicos de ap endizaje au om´a ico, p oponemos un en oque basado en el ap en-
dizaje p o undo pa a educi la dimensionalidad de los da os de exp esi´on, que emplea
ap endizaje supe isado y no supe isado pa a hace uso de odas las mues as de umo es
p esen es en una base de da os pa a esol e una a ea de clasi icaci´on en c´ance conc-
e a. Empleando la p edicci´on del sub ipo in ´ınseco de c´ance de mama como ejemplo
de a ea de clasi icaci´on en c´ance , los esul ados ob enidos mues an que el endimien o
de los en oques basados en ap endizaje p o undo y en ´ecnicas adicionales de ap en-
dizaje au om´a ico son muy simila es a la ho a de educi la dimensionalidad de los da os
de exp esi´on g´enica pa a su empleo en a eas de clasi icaci´on en c´ance . Sin emba go,
aunque algunos en oques adicionales pa ecen supe a el endimein o del en oque basado
en ap endizaje p o undo, pa a conclui cu´al es el en oque m´as e ec i o m´as abajo es
necesa io. Po o o lado, compa ando el endimien o de los di e en es modelos de ap en-
dizaje p o undo implemen ados, aunque con mucha p udencia, podemos deci que cuan o
mas p o undo el modelo mejo endimien o ob u o, mos ando el pode ep esen acional
i
de es os modelos pa a la ex acci´on de una je a qu´ıa de ep esen aciones abs ac as u iles
pa a la esolucion de a eas de clasi icaci´on.
Palab as cla es: ap endizaje p o undo, ap endizaje au om´a ico, in eligencia a i icial,
da os de exp esi´on g´enica, educci´on de la dimensionalidad, c´ance de mama, calsi icaci´on
Abs ac :
Deep lea ning has become one o he mos p omising A i icial In elligence (AI) echnolo-
gies nowadays. I has been e y success ully applied o a eas such as compu e ision o
na u al language p ocessing. Howe e , al hough he g ea ep esen a ional and p edic i e
capabili ies exhibi ed by hese models, hei easibili y o be applied o p oblems such
as gene exp ession da a analysis o cance classi ica ion, in which he numbe o inpu
a iables (N) a exceeds he numbe (M) o samples (NM), emains a challenge
ye o be sol ed. In o de o sol e his balancing p oblem, se e al adi ional machine
lea ning dimensioanli y educ ion echniques ha e been applied. Al hough hese ech-
niques scale down he inpu ea u e space, he p edic ion pe o mance o he adi ional
machine-lea ning models is mode a e, as he educed numbe o samples used o ain bo h
he dimensionali y educ ion me hods and he classi ie s does no allow hem o ex ac
he hidden pa e ns in he gene exp ession da a in a way ha imp o es he p edic ion
pe o mance signi ican ly. In o de o sol e hese p oblems and imp o e he p edic ion
abili y o he adi ional machine-lea ning models, we p opose a deep lea ning app oach
o educing he dimensionali y o gene exp ession da a, which uses bo h unsupe ised
and supe ised lea ning o make he mos o he en i e umo da a a ailable in a da abase
o sol e a conc e e cance classi ica ion ask. Using b eas cance in insic sub ype p e-
dic ion as an example o a cance clasi ica ion ask, he ob ained esul s showed ha he
pe o mances o bo h deep lea ning and adi ional machine-lea ning app oaches a e e y
simila when educing he dimensionali y o gene exp ession da a o he pu pose o cance
classi ica ion. Howe e , hough some adi ional app oaches seem o ou pe o m he deep
ii
lea ning s a egy, o conclude which is he mos e ec i e app oach mo e wo k needs o
be done. On he o he hand, compa ing he pe o mance o he di e en au oencode s,
al hough e y cau iously, we could say ha he deepe he model he be e pe omance
was ob ained, showing he ep esen a ional powe o deep lea ning o ex ac a hie a chy
o abs ac ep esen a ions use ul o sol ing classi ica ion asks.
Keywo ds: deep lea ning, machine-lea ning, a i icial in elligence, gene exp ession da a,
dimensionali y educ ion, b eas cance , classi ica ion
iii
is analyzed by he omics disciplines, such as genomics, ansc ip omics, p o eomics o
epigenomics. The in eg a ion o all his da a om a single o ganism can be e y aluable in
ields such as medicine, whe e his da a is used o ge a holis ic iew o he molecula s a e
o a pa ien , d i ing he p og ess o wha is called P4 medicine (p edic i e, p e en i e,
pe sonalized and pa icipa o y), conside ed by he expe s o be he medicine o he u u e
[23].
The con ibu ions o deep lea ning o his domain a e mainly ound in genomics (a
key a ea in P4 medicine), due o he adap a ion o he con olu ional models applied in
compu e ision o DNA sequence da a. Ins ead o p ocessing 2-D images wi h h ee colo
channels, a DNA sequence is conside ed as a 1-D sequence wi h ou channels, one o
each ype o nucleo ide (A, C, T, G) [24]. In 2015, his app oach was used by Alipanahi
e al o ind use ul mo i s in DNA sequences o p edic sequence speci ici ies o DNA-
and RNA-binding p o eins [25]. One yea la e , he same con olu ional s a egy was used
by Zhou and T oyanskaya o ind e ec i e mo i s o p edic ing he e ec s o non-coding
a ian s in DNA sequences [26].
Apa om genomics, gene-exp ession da a analysis ( ansc ip omics) is becoming one o
he mos impo an omics disciplines in P4 medicine, due o he ad en o high- h oughpu
sequencing echnologies such as RNA-Seq [27]. In a eas such as oncology, gene exp ession
da a o e s a comple ely new way o desc ibing he molecula s a e o a pa ien . As
cance is conside ed a gene ic disease, a gene exp ession sample om a pa ien (which
desc ibes he gene ic changes esponsible o he p og ession o he disease, such as he
o e -ac i i y o he ep ession o genes) con ains in o ma ion o pa amoun impo ance
o he p e en ion, diagnosis and ea men o his malignan disease. Fo example, in
b eas cance (one o he mos he e ogeneous cance s wi h many in insic sub ypes) he
in o ma ion hidden in he gene exp ession da a, when p ope ly ex ac ed, can be used o
diagnose he conc e e sub ype in a p ecise and e ec i e way [28]. An accu a e diagnosis
is ex emely impo an o he de elopmen o a pe sonalized ea men , as he molecula
and speci ic he apies as well as he p edic ed p ognosis s ongly depend on he in insic
b eas cance sub ype o he pa ien [29].
The con ibu ions o deep lea ning o gene exp ession analysis o cance p edic ion a e
ex emely sca ce. The eason being ha , al hough deep lea ning models ha e demon-
5

s a ed o be able o ex ac he hidden pa e ns in ex emely complex da a, gene ex-
p ession da a p esen some p oblems ha make he applica ion o deep lea ning models
a di icul challenge ye o be sol ed. Up o now, in all he success ul deep lea ning ap-
plica ions we ha e men ioned, he da a had spa ial o local in o ma ion ( ex sequences,
images, biological sequences, e c.); howe e , his is no he case wi h gene exp ession da a.
To make ma e s wo se, he dimensionali y o he inpu ea u e space, i.e. he numbe o
inpu ea u es (N), is ex emely high (10K-60K) in gene exp ession da ase s. Howe e ,
in clinical asks such as cance de ec ion, he numbe o a ailable samples (M) is e y
low (300-1K). This eno mous imbalance be ween he numbe o inpu ea u es and he
numbe o a ailable samples (NM) makes he lea ning p ocess ex emely di icul ,
and i is known as he cu se o dimensionali y [30], a common p oblem no only o deep
lea ning models, bu o adi ional machine-lea ning algo i hms as well.
In ac , in o de o sol e di e en cance classi ica ion asks using gene exp ession da a,
a ious adi ional machine-lea ning models ha e been applied, such as logis ic eg ession,
decision ees, suppo ec o machines, swallow a i icial neu al ne wo ks, e c. [31, 32].
Bu again, he main p oblem aced by hese algo i hms is he high dimensionali y o he
gene exp ession da a compa ed o he lack o a ailable samples (NM). To educe he
numbe o inpu ea u es, dis inc classic dimensionali y educ ion echniques ha e been
used, such as ea u e selec ion and ex ac ion me hods [33]. Al hough hese echniques
scale down he inpu ea u e space, he p edic ion pe o mance o he adi ional machine-
lea ning models is mode a e, as he educed numbe o samples used o ain bo h he
dimensionali y educ ion me hods and he classi ie s does no allow hem o ex ac he
hidden pa e ns in he gene exp ession da a in a way ha imp o es he p edic ion pe -
o mance signi ican ly. This is mainly due o a scalabili y p oblem, as, o example, when
using adi ional machine-lea ning me hods o p edic he in insic b eas cance sub ype
om gene exp ession da a, hese supe ised models canno ake ad an age o any o he
umo da a bu b eas cance , using only a ew hund ed samples o ain he models.
In o de o sol e hese p oblems and imp o e he p edic ion abili y o he adi ional
machine-lea ning models, we p opose a deep lea ning app oach o educing he dimen-
sionali y o gene exp ession da a, which uses bo h unsupe ised and supe ised lea ning o
make he mos o he en i e umo da a a ailable in a da abase o sol e a conc e e cance
6
classi ica ion ask, such as p edic ing he in insic b eas cance sub ype o a pa ien .
Now, we shall p oceed o e iew he s a e o he a in deep lea ning o cance de ec ion
using gene exp ession da a.
1.2.1 Deep lea ning o gene exp ession da a analysis
Al hough, as i was said be o e, he con ibu ions o deep lea ning o cance p edic ion
using gene exp ession da a a e jus s a ing o eme ge and he e a e no ye nume ous
examples, se e al wo ks a e wo h men ioning, as hey cla i y how unsupe ised deep
lea ning models (essen ially au oencode s) can be adap ed o educe he dimensionali y o
gene exp ession da a o he pu pose o cance classi ica ion. One o he mos inspi ing and
ci ed ones was done in 2013 by Fakoo e al [34]. In his wo k, hey used a combina ion o
PCA and simple au oencode a chi ec u es (spa se and wo-laye s s acked au oencode s)
o pe o m dimensionali y educ ion, and a so max ou pu laye on op o he au oencode
a chi ec u e du ing he classi ica ion s age. Al hough he au oencode s a e cons ained
by he ea u es ex ac ed by PCA, hey use a e y simple linea classi ie and only 2K
samples o aining he deep models, his is he i s wo k using deep au oencode s and
gene exp ession da a om di e en umo s du ing he ea u e lea ning s ep. In 2016,
Danaee e al used s acked denoising au oencode s o ea u e ex ac ion, and e alua ed
he ex ac ed ep esen a ions pe o ming supe ised b eas cance de ec ion [35]. E en
hough such a deep model was ained using only 1K b eas cance samples, his was he
i s ime s acked denoising au oencode s we e applied o gene exp ession da a. Finally,
in 2018, Way and G eene employed a ia ional au oencode s (VAEs) o ex ac a la en
ea u e space using 10K gene exp ession samples [36]. Though hey did no use he
ex ac ed ea u es o pe o m any cance de ec ion ask, hey showed ha he ex ac ed
ea u es ep esen ed biological signals.
In his inal p ojec , basing on hese p e ious wo ks, we will y o use deep au oencode s
o educe he dimensionali y o he gene exp ession da a, as well as a ans e lea ning
app oach ha allows us o ain he deep lea ning models using a whole da abase o umo
samples, and use he ex ac ed ea u es o pe o m cance classi ica ion asks such as he
p edic ion o b eas cance in insic sub ypes.
7
1.3 Objec i es
Thus, he main objec i e o his p ojec is o use a deep lea ning app oach o educe he
dimensionali y o gene exp ession da a, as well as analyzing i s e ec i eness when applied
o cance classi ica ion asks.
This p incipal objec i e can be di ided in o wo mo e speci ic objec i es:
1. Adap ing deep lea ning models o gene exp ession da a pa icula di icul ies. In
o de o sol e he g ea imbalance be ween he dimensionali y o he da a (N) and
he numbe o samples (NM), we will y di e en au oencode s a chi ec u es
(spa se, s acked spa se and denoising s acked) o educe he numbe o inpu ea-
u es. In addi ion, we will use a ans e lea ning app oach o inc ease he numbe
o samples used du ing he ea u e ex ac ion s age.
2. Compa ing he ob ained esul s using a deep lea ning app oach wi h he ones ob-
ained using adi ional machine-lea ning dimensionali y educ ion echniques. In
o de o compa e bo h app oaches, we will use he ex ac ed ea u es o sol e a
cance classi ica ion ask, such as he p edic ion o he b eas cance in insic sub-
ypes. To do ha , we will use h ee classic supe ised machine-lea ning algo i hms:
Logis ic eg ession, suppo ec o machines and swallow a i icial neu al ne wo ks.
1.4 Documen s uc u e
In his sec ion, we gi e a b ie desc ip ion o he s uc u e o he documen :
•Me hods: This sec ion desc ibes each o he phases o a adi ional da a-mining
me odology ollowed o ca y ou his p ojec , such as da a ex ac ion, da a p e-
p ocessing, dimesnionali y educ ion, e c. In addi ion, i con ains a desc ip ion o
he algo i hms used o pe o m all hese s ages.
•Resul s: He e, he p ojec inal esul s a e p esen ed and discussed. Special a en-
ion will be paid o he compa ison o he esul s ob ained using he deep lea ning
app oach and he adi ional machine-lea ning s a egy o educe he dimensionali y
o he da a.
8
•Conclusion: Finally, he las sec ion con ains a conclusion o he wo k bo h in
English and Spanish, wi h a inal subsec ion dedica ed o desc ibe u u e wo ks and
esea ch lines.
9
2 Me hods
In his p ojec , we compa e wo di e en s a egies o sol ing a cance classi ica ion ask
using gene exp ession da a, in ou case he p edic ion o b eas cance in insic umo
sub ypes. The wo app oaches educe he high dimensionali y o he da a in a di e en
way, one using deep lea ning algo i hms and he o he using adi ional machine-lea ning
echniques.
2.1 Deep lea ning s a egy
The i s s a egy uses deep unsupe ised lea ning models (au oencode s) o pe o m ea-
u e ex ac ion. Using a ans e -lea ning app oach, he models a e p e- ained on a la ge
compendium o gene exp ession samples, and hen ine- uned using a small da ase o
classi ica ion pu pose. Hence, we dis inguish wo phases: ea u e lea ning and classi ica-
ion lea ning.
2.1.1 Fea u e lea ning
Da a ex ac ion
To p e- ain he deep models ha educe he dimensionali y o he gene exp ession da a,
any o hese public da a sou ces can be used:
•The Cance Genome A las (TCGA) pla o m is a collabo a ion be ween he Na-
ional Cance Ins i u e (NCI) and he Na ional Human Genome Resea ch Ins i u e
(NHGRI), ha has gene a ed one o he mos comple e genomic s udies up o now,
known as PanCance A las [37]. This da a con ains mu i-dimensional omics da a
(DNA me hyla ion, gene and p o ein exp ession da a, e c.) o 33 di e en umo
ypes om 11K pa ien samples.
•The UCSC Xena po al allows o access 1521 mul i-omics da ase s om 135 di -
e en coho s. The la ges da ase o which hey p o ide access o is he TCGA-
TARGET-GTEx da ase , a da a in eg a ion om h ee di e en pla o ms in o a
10

unique da ase ee o compu a ional ba ch e ec s [38]. I comp ises 20K gene ex-
p ession samples, om which almos he 50% o hem come om cance pa ien s,
and he o he 50% om heal hy (con ol) pa ien s. This makes TCGA-TARGET-
GTEx one o he la ges and mos balanced gene exp ession da ase s, some hing
specially use ul when pe o ming cance de ec ion asks such as p edic ing whe he
a sample comes om a cance o heal hy pa ien .
•The Gene Exp ession Omnibus (GEO) is a public unc ional genomics da a epos-
i o y ha s o es and eely dis ibu es mic oa ay, nex gene a ion sequencing and
many o he high- h oughpu unc ional genomics da a p o ided by he scien i ic
communi y. Pla o ms such as ARCSh4[39], p o ide access o all he RNA-Seq
gene exp ession da a a ailable in GEO, p ocessing he da a om di e en pla -
o ms uni o mly. Conc e ely, 187,946 samples a e accessible h ough ARCSh4wi h
103,083 mouse and 84,863 human.
Due o pe o mance easons, he da ase used in his wo k o p e- ain he models is
he Pan-Cance gene exp ession da ase . Al hough ou ini ial in en ion was o use he
TCGA-TARGET-GTEx da ase , i con ains almos wice (20K) he numbe o samples
o he Pan-Cance (11K), which makes i oo la ge conside ing ou ha dwa e esou ces
(see sec ion ).
Da a p epa a ion
The o iginal Pan-Cance da ase con ains 11K samples and 60K a iables (gene an-
sc ip s). Howe e , ou ha dwa e esou ces canno p ocess so many inpu a iables (see sec-
ion ). Hence, ins ead o using he o iginal da ase , we used he da a om [36] (hence o h
called pan-cance da ase ), accessed h ough h ps://gi hub.com/g eenelab/ ybal .
This da ase con ains all he 11K samples om 33 umo ypes bu only includes he 5K
mos a iably exp essed genes, de ined by median absolu e de ia ion (MAD). In addi ion,
he uni o he gene exp ession da a is log2(FPKM + 1) ans o med RSEM alues.
Ac ually, a he han using he whole pan-cance da ase o p e- aining he models,
we spli he da a in o wo dis inc sub-da ase s: one con aining only he b eas cance
umo samples (BRCA pan-cance da ase , 1K samples) and he o he con aining he e-
11
maining samples om he es o he 32 umo ypes (non-BRCA pan-cance da ase , 10K
samples). As in ou ans e -lea ning app oach he cance classi ica ion ask we wan o
sol e is he p edic ion o he in insic b eas cance sub ype, he BRCA pan-cance da a is
only used du ing he classi ica ion lea ning phase, as i con ains he sub ypes in o ma ion,
whe eas he non-BRCA pan-cance da ase is used du ing he ea u e lea ning phase o
p e- ain he deep models in an unsupe ised way. The a ionale behind his is ha we do
wan o use o ally di e en da a o pe o m he p e- aining and he ine- uning du ing
classi ica ion lea ning phase.
On he o he hand, ega ding no maliza ion, he non-BRCA pan-cance da a is no -
malised using he s anda d cen e ing and scaling me hod (ze o mean and uni a iance).
Dimensionali y educ ion
To educe he high dimensionali y o he gene exp ession da a we use au oencode s, an
unsupe ised ea u e ex ac ion me hod.
- Theo e ical model
An au oencode , in i s simples o m, is a eed o wa d neu al ne wo k wi h h ee laye s:
an inpu , a hidden and an ou pu laye . I is an unsupe ised lea ning me hod in which
he main goal is, gi en an inpu , o econs uc an ou pu laye ep esen a ion as closely
as possible o he ini ial inpu laye ep esen a ion. This is done by aining he ne -
wo k using backp opaga ion me hod o minimize he econs uc ion e o , a unc ion ha
compu es he di e ence be ween he inpu and he ou pu .
Fo example, as i is shown in Figu e 3, gi en a se o kunlabeled aining samples
{x(1), x(2), . . . , x(k)}, whe e x(i)∈ <6, an au oencode ies o lea n a unc ion ˆx≈x[40].
The non-linea unc ion ha ans o ms he inpu in o a hidden ep esen a ion is called
encode , and can be exp essed as h(x) = (W x +b), whe e is he hidden ac i a ion
unc ion, such as sigmoid o anh, Wis he hidden weigh ma ix and bis he bias ec o
o he hidden laye . The ma ix Wis o dimensions n×d, whe e nis he dimension o he
inpu da a (numbe o uni s in he inpu laye ), and dis he dimension o he encoded
ep esen a ions (numbe o hidden uni s). On he o he hand, he non-linea unc ion
12
ha akes he hidden ep esen a ions and ans o ms hem in o he econs uc ed inpu
ep esen a ions is called decode , and can be exp essed as ˆx(h) = g(W0h+b0), whe e gis
he ou pu ac i a ion unc ion, W0is he ou pu weigh ma ix and b0is he bias ec o
o he ou pu laye . As opposed o W, he ma ix W0is o dimensions d×n.
Figu e 3: Example o a simple au oencode a chi ec u e, aken om [40].
Ha ing a hidden laye wi h ewe uni s han he inpu and he ou pu laye s (d<n),
o ces he au oencode o comp ess he inpu ep esen a ion in o a lowe dimensional
ep esen a ion, which can be econs uc ed o i s ini ial ep esen a ion. Tha is why i is
used as a dimensionali y educ ion me hod.
Cons aining he ne wo k, such as using a small numbe o hidden uni s, has demon-
s a ed o o ce he ne wo k o ex ac mo e abs ac and meaning ul ea u es in he
hidden ep esen a ions. In addi ion o educe he numbe o hidden uni s, ano he pop-
ula way o cons aining he ne wo k is using wha is called a spa si y penal y [40]. This
penal y c ea es spa se ep esen a ions, in which hidden uni s end o be inac i e mos
o he ime, i.e. close o ze o i he hidden ac i a ion unc ion is he sigmoid o ReLU
13
unc ion and close o -1 i i is he anh ac i a ion unc ion. Hence, he main e ec o
he spa se penal y is o a ou he dis ibu ed hidden (encoded) ep esen a ions and he
uni s specializa ion, as each inpu pa e n is encoded by he ac i a ion o a ela i ely
small se o hidden neu ons and each neu on esponds (ac i a es) o a small se o inpu s.
This penal y is gene ally implemen ed using L1- egula iza ion in he hidden laye , which
is added o he econs uc ion e o unc ion. Hence, i he mean squa ed econs uc ion
e o is used, he o e all loss unc ion minimized du ing he lea ning p ocedu e can be
exp essed as:
"1
m
m
X
i
||xi−ˆxi||2#+λ
n
X
j
d
X
l
|wjl|(1)
whe e mis he ba ch size, nis he numbe o inpu and ou pu uni s, dis he numbe o
hidden uni s, wjl is he weigh connec ing he inpu uni j o he hidden uni land λis
he L1- egula ize penal y. The i s e m co esponds o he inpu econs uc ion e o ,
whe eas he second e m ep esen s he L1- egula iza ion, which ends o dec ease he
magni ude o he weigh s, ac ing as a spa si y cons ain . The spa si y penal y is widely
used in image p ocessing domain, whe e i has shown o p oduce e y good esul s [41].
Ano he widely used app oach o cons aining he ne wo k is known as denoising au-
oencode s [42]. Du ing aining, noise is added o he inpu da a, and he di e ence
be ween he inpu econs uc ion and he o iginal noiseless da a is minimized using back-
p opaga ion. Hence, he goal o he ne wo k is o ob ain a hidden ep esen a ion obus
o he in oduc ion o noise in he inpu laye . In o de o be able o econs uc he inpu
co ec ly, he co up ion o he inpu da a o ces he ne wo k o ex ac mo e abs ac
and meaning ul ea u es in he hidden laye . A simple denoising au oencode a chi ec u e
can be seen in Figu e 4.
Finally, he las app oach used o o ce he ne wo k o ex ac mo e abs ac ea u es is
s acked au oencode s. This s a egy simply consis s on ”s acking” se e al au oencode s
in o a deep au oencode model, as he one shown in Figu e 5, which is he esul o
s acking wo simple au oencode s, ha ing a deep model o one inpu laye , wo hidden
encode laye s, and wo decode laye s. The hidden abs ac ep esen a ion is always he
14
Hype -pa ame e Possible alues
Numbe o eezed laye s {1,2}
Lea ning algo i hm S ochas ic G adien Descen
Lea ning a e {0.0001,0.0005,0.001}
Momen um [0.5,0.9]
D opou 2 {0,0.3,0.5}
D opou Pos {1,2}
D opou 1 {0,0.3,0.5}
Numbe o epochs [40,70]
Ba ch size [50,100]
Table 6: Fine- uning hype -pa ame e space o he s acked-spa se-denoising encode .
Fo he Logis ic Reg ession, Suppo Vec o Machine and swallow A i icial Neu al Ne -
wo k classi ica ion algo i hms, hei hype -pa ame e s a e hei possible alues a e shown
in ables 7, 8 and 9.
Hype -pa ame e Possible alues
No m penaliza ion {L1, L2}
Mul iclass {One-Ve sus-Res , Mul inomial}
Table 7: Hype -pa ame e space o he Logis ic Reg ession model.
Hype -pa ame e Possible alues
Ke nel {Radial Basis Func ion, Polynomial}
C penal y {0.1,1,10,100,1000}
Gamma [1 ×10−4, 1 ×10−1]
Polynomial ke nel deg ee [2,5]
Table 8: Hype -pa ame e space o he Suppo Vec o Machine model.
21

Hype -pa ame e Possible alues
Numbe o uni s in he hidden laye {20,40,60}
Hidden laye ac i a ion unc ion {sigmoid, anh, ReLU}
Lea ning algo i hm S ochas ic G adien Descen
Lea ning a e [0.001,2]
Momen um [0.2,0.75]
Maximum numbe o i e a ions {100,200}
Table 9: Hype -pa ame e space o he swallow A i icial Neu al Ne wo k (one-hidden
laye ) model.
2.2 T adi ional Machine-Lea ning app oach
In o de o e alua e he pe o mance o he deep lea ning app oach, we compa ed his
s a egy wi h a adi ional machine-lea ning app oach o educing he dimensionali y o
he gene exp ession da a, wi h he pu pose o pe o ming a cance classi ica ion ask.
Da a p epa a ion
The same da ase used du ing he classi ica ion lea ning s age o he p e ious sec ion
is used he e, he BRCA pan-cance da ase (1K samples) wi h he 5K mos a iably
exp essed genes as inpu a iables. Also, he same mul i-class classi ica ion ask is pe -
o med, he p edic ion o he PAM50 b eas cance in insic sub ypes.
Dimensionali y educ ion and classi ica ion
Fo educing he dimensionali y o he gene exp ession da a, we used di e en classical
echniques: wo ea u e selec ion me hods, and a ea u e ex ac ion echnique. The ea-
u e selec ion me hods co espond o il e me hods, one using ANOVA F- alues and he
o he using mu ual in o ma ion alues [33, 46]. The ea u e ex ac ion me hod is P in-
cipal Componen s Analysis (PCA), a widely used dimensionali y educ ion echnique in
22
many di e en domains [47]. Jus like when using he deep lea ning app oach, he gene
exp ession da a was educed o 100 ea u es.
To be consis en wi h wha we did when using he deep lea ning s a egy, we again used
5- old nes ed CV o e alua e he pe o mance o he models, using he a e age accu acy
measu e. This ime no ine- uning is needed, and he ea u es ex ac ed by he adi ional
dimensionali y educ ion me hods a e used by he same classi ica ion algo i hms as in
he p e ious sec ion: LR, SVM and swallow ANN. In his way, he inne CV is again
used o une some hype -pa ame e s o he classi ie s (see p e ious sec ion o de ails)
using G id-Sea ch (LR) and Randomized-Sea ch (SVM and ANN), and he ou e CV is
used o ain he bes selec ed models and e alua e hei pe o mance, by i s educing
he dimensionali y o he da a. We also s anda d scaled he da a (ze o mean and uni
a iance) be o e eeding i in o he classi ie s.
23
3 Resul s
3.1 Deep models p e- aining
As we s a ed be o e, he h ee au oencode s models we e p e- ained using he non-BRCA
pan-cance da ase . Fo uning se e al hype -pa ame e s o each a chi ec u e, we simply
spli he da a in o a aining (90% o he samples) and a alida ion se (10%), and hen
examine he loss cu es ying di e en alues con igu a ions.
In his way, when using he spa se a chi ec u e, we selec ed he alues showed in Table
10. Using hese alues, he aining and alida ion loss cu es a e he ones showed in
Figu e 6. The aining p ocess seems o be qui e s able, hough some oscilla ions a e
obse ed. Howe e , o e i ing clea ly exis s, as we can see om he dis ance be ween he
aining and alida ion loss cu es.
Hype -pa ame e Possible alues
L1 egula iza ion 1 ×10−5
Lea ning a e 0.001
Numbe o epochs 40
Ba ch size 100
Table 10: Hype -pa ame e s selec ed alues o he spa se one-hidden laye au oencode
model.
Fo he s acked-spa se model, he selec ed hype -pa ame e s alues con igu a ion is
desc ibed in Table 11. As we can see om Figu e 7, he aining p ocess seems o be
qui e s able oo, and o e i ing is again obse ed.
Finally, when using he s acked-spa se-denoising au oencode , he selec ed alues o he
hype -pa ame e s a e shown in Table 12. The loss cu es in Figu e 8 show an inc edibly
s able lea ning p ocess, wi h no obse able luc ua ion, as well as he almos o e i ing
inexis ence, some hing eally di icul o ob ain in such a deep model like his one ( i e
hidden laye s). This o e i ing educ ion in compa ison wi h he o he wo models is due
24
Figu e 6: T aining and alida ion loss cu es o he spa se a chi ec u e.
Laye Hype -pa ame e Possible alues
Hidden one (encode ) L1 egula iza ion 1 ×10−5
Hidden wo (encode ) L1 egula iza ion 1 ×10−5
Whole model
Lea ning a e 0.0005
Numbe o epochs 150
Ba ch size 200
Table 11: Hype -pa ame e s selec ed alues o he deep s acked-spa se h ee-hidden laye s
au oencode model.
o he noise used in he inpu laye o he denosing model, implemen ed using d opou ,
one o he mos e ec i e echniques o educe o e i ing in complex deep ne wo ks [11].
3.2 Classi ica ion esul s
The pe o mance o ou deep lea ning app oach in b eas cance sub ype classi ica ion is
summa ized in Table 13, whe eas he Table 14 con ains he pe o mance o he classical
machine-lea ning app oach. All he alues con ained in bo h ables ep esen he a e age
classi ica ion mul i-class accu accy ob ained acc os he 5 i e a ions o ou 5- old nes ed
C oss-Valida ion p ocedu e, and he pe o mance o he models is e alua ed in e ms o
he a e age es ACC.
25
Laye Hype -pa ame e Possible alues
Hidden one (encode ) L1 egula iza ion 1 ×10−5
Hidden wo (encode ) L1 egula iza ion 1 ×10−5
Hidden h ee (encode ) L1 egula iza ion 1 ×10−5
Whole model
Noise a io 0.15
Lea ning a e 0.0001
Numbe o epochs 200
Ba ch size 225
Table 12: Hype -pa ame e s selec ed alues o he deep s acked-spa se-denoising i e-
hidden laye s au oencode .
Au oencode Classi ie Tes ACC T ain ACC
Spa se
So max 0.867 ±0.013 0.997 ±0.003
LR 0.864 ±0.018 0.995 ±0.005
SVM 0.879 ±0.022 0.999 ±0.001
ANN 0.849 ±0.025 0.993 ±0.014
S acked-spa se
So max 0.887 ±0.012 1 ±0
LR 0.878 ±0.012 1 ±0
SVM 0.888 ±0.021 1 ±0
ANN 0.888 ±0.015 1 ±0
S acked-spa se-denoising
So max 0.903 ±0.019 1±0
LR 0.900 ±0.011 1±0
SVM 0.893 ±0.010 1±0
ANN 0.889 ±0.011 1±0
Table 13: Pe o mance o he classi ica ion algo i hms using he deep lea ning app oach
o dimensionali y educ ion. In addi ion o he ou adi ional ML classi ie s, we also
include he so max used o pe o m ine- uning.
Dimensionali y educ ion Classi ie Tes ACC T ain ACC
Ano a
LR 0.897 ±0.023 0.971 ±0.006
SVM 0.894 ±0.019 0.956 ±0.028
ANN 0.899 ±0.021 0.983 ±0.014
Mu ual-In o ma ion
LR 0.908 ±0.018 0.972 ±0.007
SVM 0.895 ±0.021 0.987 ±0.016
ANN 0.901 ±0.010 0.981 ±0.021
PCA
LR 0.902 ±0.021 0.982 ±0.011
SVM 0.906 ±0.030 0.999 ±0.001
ANN 0.905 ±0.018 0.977 ±0.014
Table 14: Pe o mance o he classi ica ion algo i hms using he adi ional machine-
lea ning app oach o dimensionali y educ ion.
26

Figu e 7: T aining and alida ion loss cu es o he s acked-spa se a chi ec u e.
Figu e 8: T aining and alida ion loss cu es o he s acked-spa se-denoising a chi ec u e.
When compa ing he di e en au oencode a qui ec u es, he deep models ou pe o m
he single hidden laye spa se au oencode using any o he ou classi ie s. Besides, he
deep s acked-spa se-denoising model ob ains he bes esul s, showing ha i s deep a chi-
ec u e and he co up ed da a used du ing aining allows he model o ex ac he mos
27
use ul ea u es o classi ica ion pu pose. Howe e , cau ion is needed, as he di e ences
be ween he 5- old CV a e age accu acies a e no s a is ically signi ican enough o d aw
ca ego ical an de ini i e conclusions. On he o he hand, compa ing he classi ica ion al-
go i hms, al hough he bes es ACC alue is ob ained when using so max classi ie and
s acked-spa se-denoising model, SVM seems o be mos consis en when combined wi h
au oencode models, being he classi ie ha ob ains he bes pe o mance when using
spa se and s acked-spa se a chi ec u es.
Analyzing he adi ional ML app oach, hough he maximum es ACC alue is ob-
ained when using mu ual-in o (and LR), PCA seems o be he mos consis en me hod.
Compa ing Table 13 and Table 14, he adi ional ML s a egy seems o ou pe o m
ou DL app oach. Howe e , he deep s acked-spa se-denoising a qui ec u e ou pe o ms
Ano a using any classi ie . Hence, om he ob ained esul s, we can say ha , hough
some adi ional app oaches (mu ual-in o and PCA) seem o ou pe o m he deep lea ning
s a egy, o conclude which is he mos e ec i e app oach mo e wo k needs o be done.
One o he main easons why DL s a egy does no seem o ou pe o m he adi ional
ML app oach is o e i ing. Compa ing ain and es ACC, he e is a much bigge
di e ence be ween hose alues in Table 13 han in Table 14. Al hough d opou echnique
was used, o he egula iza ion me hods ha p e en o e i ing may be needed in o de
o boos he pe o mance o DL models in his pa icula cance classi ica ion ask.
I is wo h men ioning ha ou ha dwa e limi a ions may has some hing o do wi h he
ac ha adi ional machine-lea ning me hods seems o ou pe o m ou deep lea ning
app oach. Fo he s acked-spa se and s acked-spa se-denoising deep a chi ec u es, we
could only execu e e y ew i e a ions o he Randomized-Sea ch p ocedu e o une he
hype -pa ame e s o he models, which could ha e been uned much mo e e ec i e using
a g ea e numbe o i e a ions, possibly imp o ing he p edic ion pe o mance o hose
models.
28
4 Conclusion
In his p ojec , we ha e ied o adap deep lea ning, one o he mos p omising and
success ul AI echnologies nowadays, o bioin o ma ics domain, in pa icula o gene ex-
p ession analaysis o cance classi ica ion. Ou main goal was o gi e a solu ion o he
eno mous imbalance be ween he numbe o inpu ea u es (N) and he numbe (M) o
umo gene exp ession a ailable samples (NM) using deep lea ning models, o he
pu pose o pe o ming b eas cance in insic sub ypes classi ica ion. In his way, we ha e
used h ee dis in ypes o au oencode s, an unsupe ised ea u e lea ning echnique, o
educe he dimenionali y (N) o he gene exp ession da a by a ac o o 50. Besides, using
a ans e -lea ning app oach, we we e able o p e- ain he models using a la ge com-
pendium o umo da a, di e en om he da a used o pe o m he cance classi ica ion
ask, and hence inc easing he po en ial numbe o samples (M) used o ain he models
in an unsupe ised way. This also allows deep lea ning models o ex ac gene ic ea u es
om a la ge umo da ase ha may be use ul o sol ing a conc e e cance classi ica ion
ask such as b eas cance sub ype p edic ion. Using a nes ed C oss-Valida ion s a -
egy, we also compa ed he pe omance o he deep lea ning app oach wi h a adi ional
machine lea ning s a egy o educing he dimenionali y o he da a.
The ob ained esul s showed ha he pe o mances o bo h deep lea ning and adi ional
machine-lea ning app oaches a e e y simila when educing he dimensionali y o gene
exp ession da a o he pu pose o b eas cance sub ype classi ica ion. Howe e , hough
some adi ional app oaches seem o ou pe o m he deep lea ning s a egy, o conclude
which is he mos e ec i e app oach mo e wo k needs o be done. On he o he hand,
compa ing he pe o mance o he di e en au oencode s, al hough e y cau iously, we
could say ha he deepe he model he be e pe omance was ob ained, showing he
ep esen a ional powe o deep lea ning o ex ac a hie a chy o abs ac ep esen a ions
use ul o sol ing classi ica ion asks.
29
4.1 Fu u e wo k
In o de o sol e he di icul ies aced by gene exp ession analysis classi ica ion asks,
di e en deep lea ning app oaches a e ye o be explo ed. Apa om ans e -lea ning,
gene a i e models such as GANs o VAEs could be used o gene a e a i icial gene ex-
p ession samples, and hence balancing he eno mous disp opo ion o inpu ea u es and
a ailable samples (NM) men ioned abo e.
On he o he hand, much mo e wo k needs o be done o exhaus i ely analyze he ea-
sibili y o he ans e -lea ning app oach used in his p ojec . We ha e used he ex ac ed
ea u es by he p e- ained models o sol e only one cance classi ica ion ask, bu he e
a e many mo e cance p edic ion asks we e gene ic ea u es ex ac ed by deep lea ning
models om a la ge compendium o umo da a could be e y aluable.
Finally, ano he o he mos unexplo ed a eas o deep lea ning is model in ep e abili y.
Al hough many e o s ha e been made, mos o deep lea ning models a e s ill conside ed
as ”black-boxes”. In a eas such as bioin o ma ics o medicine, i we wan o apply hese
models, in e p e abili y mus no be a lacking quali y, bu a cha ac e is ic.
Finally, o sum up, i should be no iced ha he wo global objec i es o he p ojec ha e
been g ea ly accomplished, and he p ojec has se ed as my i s expe ience in w i ing
and ca ying ou a scien i ic p ojec , in which many o he concep s lea n h oughou he
Bioin o ma ics deg ee ha e been success ully applied.
30
42. P. Vincen , H. La ochelle, Y. Bengio, P. A. Manzagol, Ex ac ing and composing
obus ea u es wi h denoising au oencode s. P oceedings o he Twen y- i h In e -
na ional Con e ence on Machine Lea ning, 1096–1103 (2008).
43. J. Tan, M. Ung, C. Cheng, C. S. G eene, Unsupe ised ea u e cons uc ion and
knowledge ex ac ion om genome-wide assays o b eas cance wi h denoising au-
oencode s. P oceedings o he Paci ic Symposium on Biocompu ing 20, 132–143
(2015).
44. L. Gonda a, Medical image denoising using con olu ional denoising au oencode s.
IEEE 16 h In e na ional Con e ence on Da a Mining Wo kshops (2016).
45. J. S. Pa ke e al., Supe ised isk p edic o o b eas cance based on in insic
sub ypes. Jou nal o Clinical Oncology 27, 1160–1167 (2009).
46. J. R. Ve ga a, P. A. Es ´e ez, A e iew o ea u e selec ion me hods based on mu ual
in o ma ion. Neu al Compu ing and Applica ions 24, 175–186 (2014).
47. H. Abdi, L. J. Williams, P incipal componen analysis. Wiley In e disciplina y Re-
iews: Compu a ional S a is ics 2, 433–459 (2010).
37