T abajo Fin de M´
as e
A new mul i a ia e da a analysis
model: cons ained Na¨ı e Bayes
P esen ed by:
Ma ´ıa Remedios Sille o Denamiel
Supe iso s:
D . Ra ael Blanque o B a o, Uni e sidad de Se illa
D . Emilio Ca izosa P iego, Uni e sidad de Se illa
Ex e nal Supe iso :
D . Pepa Ram´
ı ez Cobo, Uni e sidad de C´adiz
June 24, 2016
Ag adecimien os
En p ime luga quie o ag adece a los doc o es Emilio Ca izosa P iego y Ra ael Blan-
que o B a o quienes, hace poco m´as de un a˜no, me b inda on la opo unidad de conoce
qu´e es la in es igaci´on ma em´a ica, un abajo que equie e mucha dedicaci´on y cons-
ancia, las cuales me han sabido ansmi i a la pe ecci´on.
El abajo que aqu´ı se p esen a me ha pe mi ido conoce a la doc o a Pepa Ram´ı ez
Cobo, quien adem´as de olca se con ´es e, ha sido un pila undamen al pa a m´ı en es e
iempo.
Como alguien di ´ıa: “Las pe sonas con las que abajo son, sin duda, lo mejo de
es e abajo”.
A mis pad es, And ´es y Au elia, y a mi he mana Roc´ıo, ellos son mi apoyo in-
condicional y los que me han ense˜nado a lucha po lo que quie o. G acias An onio J.
Mole o po anima me siemp e y po es a siemp e ah´ı.
G acias a odos po con ia en m´ı.
3
Con en s
In oduc ion 7
1 Mo i a ion and Backg ound 9
1.1 De ini ion o a classi ie and pe o mance measu es . . . . . . . . . . . 9
1.2 O e all iew o classi ica ion me hods . . . . . . . . . . . . . . . . . . . 10
1.3 Benchma kclassi ie s............................ 11
1.3.1 Suppo Vec o Machine . . . . . . . . . . . . . . . . . . . . . . 11
1.3.2 RandomFo es ........................... 13
1.3.3 Na¨ı eBayes............................. 14
2 Na¨ı e Bayes classi ie 15
2.1 Desc ip ion ................................. 15
2.2 Discussion abou he independence hypo hesis . . . . . . . . . . . . . . 16
2.2.1 Maximal In o ma ion Coe icien (MIC) . . . . . . . . . . . . . . 17
2.3 Pa ame e s’ es ima ion me hodologies . . . . . . . . . . . . . . . . . . . 19
2.3.1 The equen is e sus he Bayesian pa adigms . . . . . . . . . . 19
2.3.2 The No mal-Gamma model . . . . . . . . . . . . . . . . . . . . 21
2.3.3 A simula ion example . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 A ealda ase example........................... 29
2.4.1 Uni a ia e and bi a ia e analysis . . . . . . . . . . . . . . . . . 29
2.4.2 Compa ison be ween he equen is and Bayesian app oaches . 30
3 The cons ained Na¨ı e Bayes 43
3.1 Mo i a ion and o mula ion o he op imiza ion p oblem . . . . . . . . 43
3.2 An algo i hm o sol e he op imiza ion p oblem . . . . . . . . . . . . . 45
3.3 Nume icalexample ............................. 46
Conclusions 51
5

In oduc ion
The classi ica ion p oblems a e in ogue due o he ac ha hey a e necessa y in
many ilds o eal li e, whe e he da ase s ga he so much in o ma ion. Some examples
ha e eal he impo ance o hese p oblems a e: he ea ly de ec ion o diseases, he
g an ing o c edi o a ce ain indi idual,. . . I seems clea ha o wo k wi h good ea-
u es, ha is, a ibu es ha a e able o dis inguish he di e en classes, is a key poin .
O e he yea s, he classi ica ion p oblems ha e been s udied and many classi ie s
ha e been de eloped. Some examples o hem a e: Suppo Vec o Machine ([5]), Ran-
dom Fo es ([3]), Na¨ı e Bayes ([6],[11],[8],[10]), be ween o he s. Once a classi ie is
de ined, he ques ion ha e e yone wan s o know is how good is a gi en classi ica ion
unc ion. In o de o sol e his ques ion, measu es o e ec i eness we e de ined ([16]).
This wo k ocuses on he Na¨ı e Bayes classi ie . A ho ough s udy o he beha io
o he Na¨ı e Bayes classi ie , he e ec o assuming independence and he in luence o
he in ol ed pa ame e s es ima ion me hods is p esen ed. Finally, as a majo con i-
bu ion o his wo k, a di e en e sion o Na¨ı e Bayes classi ie is p esen ed, in which
he es ima ion is made by imposing cons ain s on he e ec i eness measu es on he
ob ained classi ie .
The s uc u e o he wo k is desc ibed nex . In Chap e 1 an in oduc ion o
classi ica ion is made. In Chap e 2, Na¨ı e Bayes classi ica ion is explained in mo e
de ail, in addi ion o in oduce a coe icien o he calcula ion o bo h linea and
nonlinea ela ionships be ween a iables ([15], [18]). Finally, he di e en pa ame e s
es ima ion me hodologies will be compa ed. In Chap e 3 a s udy o he no el app oach
will be ca ied ou .
7
Chap e 1
Mo i a ion and Backg ound
1.1 De ini ion o a classi ie and pe o mance mea-
su es
A classi ie is a unc ion, called , ha associa es inpu a iable ec o s x= (x1, . . . , xp)∈
X o ou pu classes y∈ {C1, . . . , CK}, whe e Xis he space o a iables and xi∈R
al hough hey will be con inuous and disc e e. Mo eo e , i is conside ed ha classes
a e ca ego ical and espec i ely exclusi e. The aim o s a is ical classi ica ion is o
lea n om a labeled aining da ase o Ninpu -ou pu pai s, (xn, yn), n= 1 . . . N,
whe e xna e he indi iduals and yndeno es he class o such indi idual.
Once he classi ie has been ob ained using he aining se , i s pe o mance is mea-
su ed ([16]) by using a es da ase so as o a oid he o e - i ing. No e ha bo h
samples, aining and es , can be assumed o be gene a ed om he same popula ion.
Le he ue class o a gi en indi idual be deno ed xand he class o he same
indi idual indica ed by he classi ie be deno ed y. Al hough i is expec ed ha hey
a e he same o as many indi iduals as possible in he es da ase , ne e heless, he
classi ie will make e o s. The p obabili ies ha he classi ie will e u n co ec esul s
on each o he classes a e es ima ed in a equen is manne by di iding he numbe
o igh ly classi ied indi iduals o a gi en class by he o al numbe o indi iduals in
ha class in he es da ase .
The p obabili y P(y=Ck|x=Ck) is unde s ood as he p obabili y ha he
classi ie designa es class Ck, gi en ha he indi idual belongs o class Ck. Now,
P(y=Ck|x=Cl), k, l ∈ {1, . . . , K},k6=l, shows he p obabili y ha he classi ie
ails in classi ying an indi idual ac ually om class Clas an indi idual in class Ck.
9
independen on he class), in p ac ice, he in e es is in compu ing he nume a o ,
which can be w i en as he join dis ibu ion
p(x, Ck) = p(x1|x2. . . xn, Ck)p(x2|x3. . . xn, Ck). . . p(xn|Ck)π(Ck).(2.3)
The key assump ion o he NB, which makes i a so ac able classi ie e en o la ge
alues o p, is he independence o he ea u es condi ioned o he class, which implies
ha (2.3) can be simpli ied o
p(x, Ck) = π(Ck)
p
Y
i=1
p(xi|Ck).
Finally, he p obabili ies o in e es (2.2) will be compu ed acco ding o
p(Ck|x)∝π(Ck)
p
Y
i=1
p(xi|Ck).
No e ha in o de o implemen he NB classi ie , a p io dis ibu ion π(·) o he
class as well as a p obabili y dis ibu ion o he ea u es condi ioned o he class Xi|Ck
need o be selec ed. Common choices o he las include he No mal and Mul inomial
dis ibu ions, o he con inuous and disc e e cases, espec i ely. Conce ning he choice
o he p io , i can be based on he esea che ’s p e ious knowledge o he p oblem o ,
in he case o lack o p io knowledge, i can be se equal o all classes ( ha is,
π(Ck)=1/K, o all k= 1, . . . , K).
2.2 Discussion abou he independence hypo hesis
As p e iously commen ed, he independence assump ion in he NB no ably simpli ies
he compu a ion o he condi ional p obabili ies as in (2.2). Howe e , his assump ion
is gene ally no ue in p ac ice and iola ed o many eal da ase s. The aim o his
sec ion is o discuss how his a ec s he pe o mance o he classi ie . As a oy example,
conside h ee ea u es X1, X2, X3such ha X1and X2a e independen , bu X3is
highly co ela ed wi h X1. The assump ion o independence leads o decomposing
he join densi y as he p oduc o he h ee ma ginals, and he e o e he in luence
o X1(≃X3) is di e en han i should be. Hence, i is na u al o hink ha such
miscalcula ions may possibly de e io a e he classi ie ’s pe o mance. Howe e , he NB
has p o en o be compa able o supe io o many well-known classi ica ion al e na i es.
In [6] his ac is explained as ollows:
“...al hough he indi idual class densi y es ima es may be biased, his bias
migh no hu he pos e io p obabili ies as much, especially nea he de-
cision egions. In ac , he p oblem may be able o wi hs and conside able
bias o he sa ings in a iance such a nai e assump ion ea ns...”
16

Also, [8] p o ides ano he possible explana ion abou he o e pe o mance o he NB,
which is based on he ea u es’ selec ion p ocess. Due o he measu emen o a iables is
o en qui e expensi e ( o example, in biomedical con ex s as me abolomics), a p e ious
selec ion s ep is unde aken in o de o elimina e edundan in o ma ion. The esul
o such selec ion is a se o a iables which end o be weakly co ela ed.
Nex , we explo e mo e in dep h he e ec o dependen ea u es on he NB pe o -
mance ia a nume ical example. Assume ha he numbe o classes is only wo (posi i e
and nega i e), and ou ea u es (M1, M2, M3 and M4) a e simula ed acco ding o a
mul i a ia e No mal dis ibu ion wi h means ec o s gi en by
µ+= (2,3,4,1), µ−= (3,6.5,2,3.5)
and co a iance ma ix de ined by
M1M2M3M4
M1
M2
M3
M4




1.0 0.9 0 0
0.9 1.0 0 0
0 0 1.0 0
0 0 0 1.0




No e ha ea u es a e linea ly independen , excep o M1 and M2 which a e highly
co ela ed. In o de o ain he NB classi ie (whe e No mal condi ional ma ginals
a e assumed), a sample o size equal o 1000 is used. Fo alida ion o he me hod, a
sample o 300 obse a ions is conside ed. In bo h cases, he p opo ion o he classes is
equal. To implemen he NB, he ou ine Nai eBayes om he lib a y klaR is used
(see he Appendix o he de ails). Table 2.1 shows he pe o mance alues (co ec
classi ica ion a es) o bo h classes, o all possible combina ions o ea u es.
F om he able, i can be obse ed how he esul s unde he selec ion (M2, M3, M4)
( ha is, when one dependen a iable is emo ed) a e be e han using he comple e
in o ma ion p o ided by he ou a iables. This ac poin s owa ds he impo ance o
a p ope ea u e selec ion me hod ha disca ds noisy in o ma ion. Because o he in-
dependence assump ion in he NB, such a ibu es’ selec ion p ocess should p oduce a
inal se o independen ea u es wi h high signi icance. T adi ionally, he Pea son co -
ela ion coe icien has been used o measu e he (linea ) dependence be ween andom
a iables. Howe e , o he ype o co ela ions (nonlinea ) may be p esen in he da a
and a e igno ed by he Pea son coe icien . The nex sec ion in oduces an index, on
which such ea u e selec ion p ocess migh be based, ha akes in o accoun nonlinea
associa ions.
2.2.1 Maximal In o ma ion Coe icien (MIC)
The Maximal In o ma ion Coe icien (MIC) (see [15],[18]) measu es nonlinea ela ion-
ships among andom a iables. I s de ini ion a ises om ha o he concep o mu ual
17
Combina ion CCR Posi i e Class CCR Nega i e Class
M1 0.741 0.708
M2 0.954 0.96
M3 0.838 0.829
M4 0.888 0.899
M1, M2 0.935 0.952
M1, M3 0.865 0.896
M1, M4 0.905 0.895
M2, M3 0.968 0.993
M2, M4 0.973 0.967
M3, M4 0.935 0.959
M1, M2, M3 0.942 0.979
M1, M2, M4 0.966 0.948
M1, M3, M4 0.953 0.947
M2, M3, M4 0.987 0.993
M1, M2, M3, M4 0.974 0.98
Table 2.1: Resul s o each combina ion o a iables.
in o ma ion (MI) (see, [9]), which quan i ies he in o ma ion abou one a iable X ha
is p o ided by a di e en a iable Y. The MI coe icien pe o ms by explo ing i he
p oduc s o ma ginal dis ibu ion p(X)p(Y) is simila o he join dis ibu ion p(X, Y ).
Speci ically, he MI coe icien be ween wo disc e e a iables Xand Yis de ined as
MI(X, Y ) = X
y∈YX
x∈X
p(x, y) log p(x, y)
p(x)p(y),
whe e p(x) and p(y) deno e he p obabili y mass unc ions o Xand Y, and p(x, y) is
he join p obabili y mass unc ion o Xand Y.
Simila ly, o he con inuous case, he coe icien is gi en by
MI(X, Y ) = ZYZX
p(x, y) log p(x, y)
p(x)p(y)dxdy,
whe e now p(·) e e s o a densi y unc ion.
The MI coe icien p esen s wo main p oblems. The i s is om a compu a ional
iewpoin , since i s calcula ion depends on a wo dimensional smoo hing. The second
p oblem is ha i is no bounded, which makes i s in e p e a ion di icul . A e mo e
han 50 yea s o de elopmen o he MI index, he MIC coe icien has been p oposed
o o e come he d awbacks o he MI. The MIC cap u es he ela ionships be ween
wo a iables using a g id on he sca e plo o hese wo a iables. To calcula e he
18
MIC, all possible g ids a e explo ed up o a maximal g id esolu ion (depending on
he sample size), and o each pai o in ege alues (x, y), he highes possible MI
ealizable by any x-by-yg id is compu ed. Then, such MI alues a e no malized in
he in e al [0,1] in o de o gua an ee he compa ison be ween all g ids. To calcula e
he MIC coe icien , i s he ma ix M= (mx,y) is de ined whe e mx,y is he la ges
no malized mu ual in o ma ion ob ained by any x-by-yg id, so ha he maximum
alue in Mwill be he alue o he MIC coe icien . Fo mally, le Ga g id and le IG
be he mu ual in o ma ion o he p obabili y dis ibu ion induced on he boxes o G,
wi h he p obabili y o a box p opo ional o he numbe o poin s d opping inside he
box. The (x, y)- h en y o Mis deno ed by mx,y, de ined as
mx,y = max IG/log min{x, y},
whe e his maximum is calcula ed o e all x-by-yg ids G. The MIC index is de ined
as he maximum o mx,y be ween all o de ed pai s (x, y) sa is ying xy < B, whe e Bis
dependen on he sample size. No e ha he elemen s o M ake alues be ween 0 and
1, and consequen ly, he MIC does, oo. Also, he MIC index sa is ies he symme y
p ope y, ha is, MIC(X, Y ) = MIC(Y, X), because o he symme y o he MI
coe icien and because o IGdepends on he ank o de o he da a. Finally, i should
be no ed ha o compu e M, i is necessa y o op imize o e all g ids, so i seems
clea ha compu ing ools will be need o calcula e he MIC alue. In pa icula , he
so wa e R compu es he MIC coe icien h ough he command mine, included in he
lib a y mi e a (see he Appendix o he de ails).
2.3 Pa ame e s’ es ima ion me hodologies
As desc ibed in Sec ion 2.1., in o de o implemen he NB classi ie , a p obabili y
dis ibu ion o he ea u es condi ioned o he class needs o be selec ed,
Xi|Ck∼Fθi,k (x).
Once such model is se , hen i needs o be es ima ed h ough sample da a. Th oughou
his wo k, i will be assumed ha Xi|Ck ollows a No mal dis ibu ion wi h unknown
mean and a iance deno ed by θi,k =µi,k, σ2
i,k. In nex sec ion we explo e wo possible
al e na i es o es ima ing he model’s pa ame e s.
2.3.1 The equen is e sus he Bayesian pa adigms
Two main app oaches can be conside ed o s a is ical in e ence: he equen is (o
classic) and he Bayesian one. The i s app oach unde akes es ima ion jus by aking
in o accoun he obse ed sample da a and, basing on he concep o equency, i
19
de i es poin es ima es o he model pa ame e s by maximizing he likelihood unc ion.
The e o e, unde a Gaussian likelihood, he equen is app oach gi es as pa ame e s’
poin s es ima es he sample mean and he sample a iance.
The second app oach conside s ins ead he possible p io knowledge ha he e-
sea che possesses abou he p oblem, and his p e ious knowledge is upda ed ( ia he
Bayes o mula) by he likelihood unc ion once da a a e obse ed. The use o he p io
knowledge implies assigning a p obabili y dis ibu ion ( he so-called p io dis ibu ion)
o he model pa ame e s, which a e ea ed as andom a iables (ins ead o unique,
unknown alues as in he equen is app oach). Such p io dis ibu ion will ans o m
(using he likelihood unc ion) in o he pos e io dis ibu ion, which is he objec o
in e es o Bayesian s a is icians. Fo mally, he Bayesian in e ence app oach pe o ms
as ollows. Suppose ha be o e he expe imen is unde aken, ou p io dis ibu ion
desc ibing he model pa ame e θis π(θ).The da a a e coming om an assumed model
(likelihood) which depends on he pa ame e and is deno ed by (x|θ).Bayes heo em
upda es he p io π(θ) o he pos e io by accoun ing o he da a x,
π(θ|x) = h(x, θ)
m(x)= (x|θ)π(θ)
m(x),
whe e m(x) is a no malizing cons an , m(x) = RΘ (x|θ)π(θ)dθ. Some o he ad an ages
o he Bayesian s a is ics a e poin ed ou nex :
•The unce ain y is exp essed ia he p obabili y dis ibu ion. The s a is ical
in e ence ollows a concep ually simple ecipe embodied in Bayes’ heo em.
•A ailable p io in o ma ion is cohe en ly inco po a ed in o he s a is ical model
desc ibing he da a.
•The FDA (US Food and D ug adminis a ion) ecommends he use o a Bayesian
me hodology in he design and analysis o clinical ials o medical de ices. Some
o he easons a e:
–Valuable p io in o ma ion is o en a ailable.
–The use o p io in o ma ion may alle ia e he need o a la ge sized ial.
–Bayesian me hods allow o g ea lexibili y in dealing wi h missing da a.
–Bayesian models acili a e me a-analysis.
Fo a de ailed desc ip ion o he Bayesian pa adigm, we e e he eade o [19].
In his wo k, because o he e sa ili y ha p io i dis ibu ions p o ide, we will
assume a Bayesian amewo k. The nex sec ion in oduces he No mal-Gamma model,
as a use ul ool o Bayesian in e ence o he No mal dis ibu ion.
20
2.3.2 The No mal-Gamma model
Conside a pai o andom a iables (X, Y ) such ha he condi ional dis ibu ion X|Y
is no mal
X|Y∼Nµ, 1
kY .
Also, Yis dis ibu ed acco ding o a Gamma p obabili y model,
Y|(α, β)∼Ga(α, β).
Then, i is said ha he pai (X, Y ) ollows a No mal-Gamma dis ibu ion, no ed
by
(X, Y )∼NG(µ, k, α, β).
The join p obabili y densi y unc ion can be ound as
(x, y |µ, k, α, β) = βα√k
Γ(α)√2πyα−1
2e−βy−ky(x−µ)2
2.(2.4)
Figu e 2.1 depic s he p e ious densi y unc ion o an asso men o pa ame e s’ alues.
Some p ope ies o he model a e as ollows. Conce ning he expec ed alues o he
andom a iables, hey a e gi en by
E(X) = µ, E(Y) = α
β.
Rega ding he a iance alues, hey can be ound as
V(X) = β
k(α−1), V (Y) = α
β2.
Finally, i can be p o en ha (2.4) is unimodal wi h he maximum a ained a µ, α−0.5
β.
The use ulness o he No mal-Gamma model in Bayesian s a is ics is ha i is a
conjuga e model. This implies ha , o a No mal likelihood (like he case conside ed
in his wo k), a No mal-Gamma p io ans o ms in o a No mal-Gamma pos e io
dis ibu ion, and he e o e, he pos e io does no need o be nume ically calcula ed.
This ac is s a ed by he nex Theo em (see [12]).
Theo em 2.3.1 Le X ollow a No mal dis ibu ion, X∼N(µ, σ2)and le x=
(x1, . . . , xn)deno e a simple andom sample o X. Assume ha µand σ2 ollow a
No mal-Gamma p io dis ibu ion, ha is
µ, 1
σ2∼NG(µ0, k0, α0, β0).
21

(a) NG(µ= 0.1, k = 2, α = 1, β = 1) (b) NG(µ= 0.1, k = 2, α = 3, β = 1)
(c) NG(µ= 0.1, k = 2, α = 5, β = 1)
(d) NG(µ= 0.1, k = 2, α = 5, β = 3)
Figu e 2.1: The p obabili y densi y unc ion o di e en No mal-Gamma models
22
Then, he pos e io dis ibu ion o Xis also a No mal-Gamma dis ibu ion, gi en by
µ, 1
σ2|x∼NG(µn, kn, αn, βn),
whe e
µn=k0µ0+n¯
x
k0+n
kn=k0+n
αn=α0+n
2
βn=β0+1
2
n
X
j=1
(xj−¯
x)2+k0n(¯
x−µ0)2
2(k0+n).
P oo :
The likelihood unc ion is
l(x|µ, σ2) =
n
Y
i=1
p(xj|µ, σ2) =
1
√2πσ2n
exp (−1
2σ2
n
X
j=1
(xj−µ)2).(2.5)
Le λdeno e he so-called p ecision, λ=1
σ2. Then
l(x|µ, σ2) = 1
(2π)n/2λn/2exp (−λ
2
n
X
j=1
(xj−µ)2).(2.6)
Le ¯
xand s2deno e he empi ical mean and a iance:
¯
x=1
n
n
X
j=1
xj
s2=1
n
n
X
j=1
(xj−¯
x)2.
Since
n
X
j=1
(xj−¯
x)(µ−¯
x) = (µ−¯
x) n
X
j=1
xj!−n¯
x!= (µ−¯
x)(n¯
x−n¯
x)=0.
23
hen, i is possible o ew i e he exponen ial e m as ollow
n
X
j=1
(xj−µ)2=
n
X
j=1
[(xj−¯
x)−(µ−¯
x)]2=
=
n
X
j=1
(xj−¯
x)2+
n
X
j=1
(¯
x−µ)2−2
n
X
j=1
(xj−¯
x)(µ−¯
x) =
=ns2+n(¯
x−µ)2(2.7)
Assume ha (µ, λ) ollows a No mal-Gamma conjuga e p io dis ibu ion, hen
(µ, λ|µ0, k0, α0, β0) = 1
Γ(α0)
βα0
02π
k01
2
λα0−1
2exp −λ
2[k0(µ−µ0)2+ 2β0],
whe e µ0, k0, α0, β0a e assumed o be known. Hence, he pos e io densi y is
(µ, λ|x)∝ (µ, λ|µ0, k0, α0, β0)p(x|µ, λ)
∝λ1
2e−(k0λ)(µ−µ0)2
2λα0−1e−β0λλn
2e−(λ
2)Pn
j=1(xj−µ)2
F om (2.7) n
X
j=1
(xj−µ)2=n(µ−¯
x)2+
n
X
j=1
(xj−¯
x)2.
Finally, i can be shown ha
k0(µ−µ0)2+n(µ−¯
x)2= (k0+n)(µ−µn)2+k0n(¯
x−µ0)2
k0+n,
whe e
µn=k0µ0+n¯
x
k0+n.
The e o e,
k0(µ−µ0)2+
n
X
j=1
(xj−µ)2=k0(µ−µ0)2+
n
X
j=1
(xj−¯
x)2=
= (k0+n)(µ−µn)2+k0n(¯
x−µ0)2
k0+n+
n
X
j=1
(xj−¯
x)2.
Thus,
24
(µ, λ|x)∝λ1
2e−λ
2(k0+n)(µ−µn)2λα0+n
2−1e−β0λe−(λ
2)Pn
j=1(xj−¯
x)2e−(λ
2)k0n(¯
x−µ0)2
k0+n.
In conclusion, he pos e io densi y ollows a No mal-Gamma dis ibu ion
p(µ, λ|x) = NG(µ, λ|µn, kn, αn, βn)
wi h
µn=k0µ0+n¯
x
k0+n
kn=k0+n
αn=α0+n
2
βn=β0+1
2
n
X
j=1
(xj−¯
x)2+k0n(¯
x−µ0)2
2(k0+n).

2.3.3 A simula ion example
He e, we es ima e om a Bayesian iewpoin and using he No mal-Gamma model
desc ibed in he p e ious sec ion, he p obabili y densi y in a Gaussian NB classi ica ion
con ex . The model pa ame e s a e es ima ed by maximizing he pos e io densi y
(Maximum a pos e io i es ima es). In his case, he pa ame e s a e No mal-Gamma
dis ibu ed a p io i, and consequen ly, acco ding o Theo em 2.3.1, hey ollow he
same p obabili y model (wi h di e en pa ame e s) a pos e io i.
The maximum a pos e io i es ima e
The maximum a pos e io i (MAP) es ima es a e pa ame e s’ poin es ima es ob ained
as he mode o he pos e io dis ibu ion. The e o e, once he pos e io dis ibu ion is
known, hen i needs o me maximized. He e, a simpli ica ion o he pos e io densi y
is ound, unde he No mal-Gamma se ing. Assume i s ha ea u e Xicondi ioned
o he class Ckis Gaussian dis ibu ed
Xi|Ck∼N(µi,k, σ2
i,k),
25
Sepal.Leng h
2.0 2.5 3.0 3.5 4.0 0.5 1.0 1.5 2.0 2.5
4.5 5.5 6.5 7.5
2.0 2.5 3.0 3.5 4.0
Sepal.Wid h
Pe al.Leng h
1234567
4.5 5.5 6.5 7.5
0.5 1.0 1.5 2.0 2.5
1234567
Pe al.Wid h
I is Da a
Figu e 2.3: Sca e plo s o each pai o a iables in he I is da ase
32

SL SW PL PW
SL
SW
PL
PW




0.99 0.66 0.23 0.19
0.66 1 0.3 0.22
0.23 0.3 1 0.14
0.19 0.22 0.14 0.9




SL SW PL PW
SL
SW
PL
PW




0.99 0.40 0.62 0.29
0.40 0.99 0.22 0.3
0.62 0.22 1 0.28
0.29 0.3 0.28 1




SL SW PL PW
SL
SW
PL
PW




0.99 0.36 0.60 0.50
0.36 1 0.40 0.38
0.6 0.39 1 0.52
0.50 0.38 0.52 0.99




No e om he MIC alues how he hypo hesis o independence becomes un ealis ic in
his example (especially o some pai s o a iables).
Some de ails a e gi en nex , ela ed o how he nume ical expe imen will be con-
duc ed. Fi s , all possible combina ions o a iables will be conside ed. Fo each
combina ion, we will p oceed as ollows.
•The ull da ase is sepa a ed in aining da ase and alida ion samples wi h
samples sizes gi en by he Table 2.7.
Se osa Vi ginica Ve sicolo
T aining 35 35 35
Valida ion 15 15 15
Table 2.7: Dis ibu ion o samples in aining and alida ion.
•In o de o ge eliable esul s and o ensu e ha he esul s do no depend on
a ce ain choice o he aining se and alida ion se , he p e ious s ep will be
epea ed 10 imes in a andom way, so ha he inal pe o mance measu e will
be he a e age/median o he measu es o each old on he alida ion se . Then,
33
–in he case o Classic ( equen is ) NB classi ie , as he pa ame e s es ima es
a e he sample mean and sample a iance, we ge hem om he aining
sample. Once hese es ima ions a e se , hen he co ec classi ica ion a e
o each class will be calcula ed on he alida ion da ase . The R command
Nai eBayes will be used o his case.
–in he case o he Bayesian NB classi ie , whe e he pa ame e s es ima es
a e ound as he solu ion o he maximiza ion o he objec i e unc ion (2.9)
using he aining sample, hen he co ec classi ica ion a e o each class
will be calcula ed on he alida ion da ase . He e, simila ly as in Sec ion
2.3., he R command nmkb will be used o maximize he pos e io densi y.
Resul s
We ocus now on he classi ica ion a es unde bo h es ima ion me hods, shown by Ta-
bles 2.8-2.11. In iew o he esul s ob ained, i seems ha , o he I is da ase , in mos
o a iable se s he a e age beha io is mo e o less simila unde bo h app oaches, al-
hough he equen is es ima ion o pa ame e s o he NB classi ie ge s mo e balanced
alues o co ec classi ica ion a e o he h ee classes, while he Bayesian In e ence o
he pa ame e s e u ns some cases ha a e mo e unbalanced in his aspec . As ano he
obse a ion, he bes accu acy has been ob ained, in bo h cases, o he combina ion
Pe al Wid h and Pe al Leng h. I is impo an o no e ha hese a e independen in
he h ee classes, because o he alue 0.52 is he highes alue ha hey ake. This
clea ly poin s ou o he al eady commen ed impo ance o he a ibu es selec ion
p ocess.
Figu es 2.4 and 2.5 depic he boxplo s o he es ima ions o he p ecision pa am-
e e unde he equen is and Bayesian app oaches, espec i ely, o Pe al Leng h and
Pe al Wid h a iables. No e om he i s igu e how he classic me hod leads o mo e
a iable esul s. No e also, how he Bayesian one seems mo e obus and i s esul s a e
no e y a ec ed by he choice o he s a ing poin in he maximiza ion algo i hm.
Figu es 2.6-2.7 a e he analogous o he p e ious ones o he means. In his case, bo h
es ima ion me hods look mo e simila .
We ound i o in e es o isually show how he p io densi y ans o med in o he
pos e io . Figu es 2.8-2.9 depic such esul s o he case o he Pe al Leng h a iable
condi ioned o he Se osa class. E en hough he densi ies belong o he same amily
o dis ibu ions i is clea he change in he modes.
34
Figu e 2.4
Figu e 2.5
35
Figu e 2.6
Figu e 2.7
36
Figu e 2.8: P io densi y o he Gaussian pa ame e s o he Pe al Leng h a iable
condi ioned o he class Se osa.
37

Figu e 2.9: Pos e io densi y o he Gaussian pa ame e s o he Pe al Leng h a iable
condi ioned o he class Se osa.
38
Sepal Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 86.66 62 62.67 70.44
Median 86.67 60 66.67 71.11
Va ia ion coe icien 0.14 0.18 0.18 0.08
Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 71.33 42.67 56 56.67
Median 73.33 43.33 56.67 57.78
Va ia ion coe icien 0.13 0.17 0.22 0.1
Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 92 93.33 95.11
Median 100 93.33 93.33 94.44
Va ia ion coe icien 0 0.75 0.07 0.02
Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 98 91.33 98 95.78
Median 100 90 100 95.56
Va ia ion coe icien 0.03 0.06 0.03 0.03
Sepal Leng h & Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 98.67 63.33 74.67 78.89
Median 100 60 76.67 80
Va ia ion Coe icien 0.03 0.14 0.14 0.04
Sepal Leng h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 80 87.33 89.11
Median 100 86.67 86.67 87.78
Va ia ion coe icien 0 0.16 0.12 0.04
Sepal Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 99.33 91.33 95.33 95.33
Median 100 90 93.33 95.56
Va ia ion coe icien 0.02 0.06 0.03 0.02
Sepal Wid h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 82 89.33 90.44
Median 100 83.33 93.33 91.11
Va ia ion coe icien 0 0.13 0.1 0.04
Sepal Wid h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 99.33 89.33 94 94.22
Median 100 86.67 96.67 94.44
Va ia ion coe icien 0.02 0.08 0.07 0.03
Table 2.8: Classic Na¨ı e Bayes.
39
Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 93.33 96.67 96.67
Median 100 93.33 100 96.67
Va ia ion coe icien 0 0.07 0.05 0.03
Sepal Leng h & Sepal Wid h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 72.67 87.33 86.67
Median 100 80 86.67 86.67
Va ia ion coe icien 0 0.15 0.12 0.05
Sepal Leng h & Sepal Wid h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 91.33 88.67 93.33
Median 100 90 86.67 94.44
Va ia ion coe icien 0 0.06 0.09 0.03
Sepal Leng h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 93.33 93.33 95.56
Median 100 93.33 93.33 95.56
Va ia ion coe icien 0 0.07 0.06 0.02
Sepal Wid h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 92.67 96.67 96.44
Median 100 90 100 96.67
Va ia ion coe icien 0 0.07 0.05 0.03
Sepal Leng h & Sepal Wid h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 92.67 94.67 95.78
Median 100 90 93.33 95.56
Va ia ion coe icien 0 0.07 0.04 0.02
Table 2.9: Classic Na¨ı e Bayes
40
Sepal Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 44 25.33 98 55.78
Median 46.67 26.67 100 56.67
Va ia ion coe icien 0.3 0.35 0.05 0.09
Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 98 0 40 46
Median 100 0 36.67 44.44
Va ia ion coe icien 0.03 0 0.02 0.06
Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 83.33 98.67 94
Median 100 86.67 100 94.44
Va ia ion coe icien 0 0.14 0.03 0.04
Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98 69.33 89.11
Median 100 100 73.33 88.89
Va ia ion coe icien 0 0.03 0.12 0.03
Sepal Leng h & Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 97.33 42 91.33 76.89
Median 100 40 93.33 75.56
Va ia ion coe icien 0.04 0.2 0.08 0.05
Sepal Leng h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 94 86.67 93.56
Median 100 93.33 86.67 93.33
Va ia ion coe icien 0 0.07 0.11 0.03
Sepal Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 60 100 86.67
Median 100 60 100 86.67
Va ia ion coe icien 0 0.1 0 0.02
Sepal Wid h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 99.33 62.67 87.33
Median 100 100 63.33 87.78
Va ia ion coe icien 0 0.02 0.3 0.07
Sepal Wid h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 90 98 96
Median 100 86.67 100 95.56
Va ia ion coe icien 0 0.07 0.03 0.03
Table 2.10: Na¨ı e Bayes and Bayesian In e ence.
41
Sepal Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean - - - -
Median - - - -
Va ia ion coe icien - - - -
Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean - - - -
Median - - - -
Va ia ion coe icien - - - -
Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 96 88 94.67
Median 100 100 86.67 93.33
Va ia ion coe icien 0 0.07 0.08 0.02
Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98 69.33 89.11
Median 100 100 73.33 88.89
Va ia ion coe icien 0 0.03 0.12 0.03
Sepal Leng h & Sepal Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean - - - -
Median - - - -
Va ia ion coe icien - - - -
Sepal Leng h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98 76.67 91.56
Median 100 100 80 93.33
Va ia ion coe icien 0 0.03 0.2 0.05
Sepal Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 87.33 96.67 60.67 81.56
Median 90 100 66.67 82.22
Va ia ion coe icien 0.16 0.05 0.18 0.07
Sepal Wid h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98.67 72.67 90.44
Median 100 100 73.33 91.11
Va ia ion coe icien 0 0.03 0.25 0.06
Sepal Wid h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 92 98 63.33 84.44
Median 100 100 66.67 85.56
Va ia ion coe icien 0.19 0.03 0.3 0.09
Table 3.1: Cons ained Na¨ı e Bayes and Bayesian In e ence.
48

Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98 88 95.33
Median 100 100 90 95.56
Va ia ion coe icien 0 0.05 0.13 0.03
Sepal Leng h & Sepal Wid h & Pe al Leng h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 96 72 89.33
Median 100 96.67 73.33 88.89
Va ia ion coe icien 0 0.05 0.22 0.06
Sepal Leng h & Sepal Wid h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 90.67 92.67 54.67 79.33
Median 96.67 93.33 56.67 78.89
Va ia ion coe icien 0.15 0.11 0.26 0.05
Sepal Leng h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 98 58.67 85.56
Median 100 100 60 85.56
Va ia ion coe icien 0 0.03 0.29 0.07
Sepal Wid h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 99.33 98 73.33 90.22
Median 100 100 70 90
Va ia ion coe icien 0.02 0.05 0.2 0.05
Sepal Leng h & Sepal Wid h & Pe al Leng h & Pe al Wid h
CCR Se osa CCR Vi ginica CCR Ve sicolo Accu acy
Mean 100 96.67 65.33 87.33
Median 100 100 63.33 86.67
Va ia ion coe icien 0 0.05 0.28 0.06
Table 3.2: Cons ained Na¨ı e Bayes and Bayesian In e ence.
49
50
Conclusions
Th ee main conclusions eme ge om he s udy on he NB classi ie ha has been ca -
ied ou in his wo k. Fi s , NB classi ie assumes independence o a iables. When
i is applied wi h dependen a iables, he ull se o a iables may no yield he bes
esul s in e ms o accu acy. This is illus a ed by he example o he I is da ase .
In ac , Fea u e Selec ion, ha has no been conside ed in his wo k beyond he com-
ple e enume a ion o se o a iables in he same example, is concluded o be necessa y.
Ano he main conclusion is ha he di e en pa ame e s es ima ion me hods in-
duce, o cou se, di e en classi ie s. This wo k does no ha e as an objec i e o indica e
which one is igh (i he equen is app oach, i he Bayesian app oach), bu o show
wi h an example ha di e en me hods induce di e en classi ie s. Al hough, ega d-
ing he Bayesian app oach, i emains o be done an exhaus i e s udy o he sensi i i y
p io s, because di e en a p io i es ima es will lead o di e en es ima es alues and
he e o e o di e en classi ie s.
The no el y o his wo k is he inclusion o cons ain s on he classical Bayesian
app oach. These cons ain s ha e p o ided a di e en classi ie , wi h he di e ence ha
his new app oach allows o con ol he pe o mance measu es, which is undamen al
in many ealwo ld classi ica ion p oblems.
51
52
Appendix: R ou ines
Na¨ı e Bayes ou ines
The language and en i onmen R (see [14]) p o ides he ollowing wo packages ela ing
o Na¨ı e Bayes classi ie :
•The package e1071, ha p o ides he unc ions
–nai eBayes: o pe o m Na¨ı e Bayes classi ica ion.
–p edic .nai eBayes: o ob ain he p edic ions.
•The package klaR ha has he ollowing unc ion o pe o m Na¨ı e Bayes clas-
si ica ion and o p edic
–Nai eBayes
–p edic .Nai eBayes
In nex subsec ions, le us explain in mo e de ail he p e ious unc ions.
Package “e1071”
Func ion nai eBayes can be called in wo di e en ways, ha is, wi h di e en ype o
a gumen s. The i s one would be:
nai eBayes(x, y, laplace=0, subse , na.ac ion=na.pass)
•x: Nume ical ma ix o da a- ame o nume ic a iables and/o ca ego ical.
•y: Vec o o classes.
•laplace: Laplace smoo hing pa ame e o imp o e he es ima es o he p obabil-
i ies in he case o ca ego ical da a (by de aul , i does no apply).
53

•subse : Vec o o indices speci ying ins ances o he sample lea ning o use.
•na.ac ion: Ac ion o pe o m in he p esence o missing alues (NA).
And he second one is:
nai eBayes( o mula, da a, laplace=0, subse , na.ac ion=na.pass)
• o mula: Fo mula in he way class ∼x1+x2+. . .
•da a: Lea ning sample da a- ame o con ingency able wi h he equencies
esul ing om he coun .
•laplace,subse ,na.ac ion: As be o e.
On he o he hand, he unc ion nai eBayes e u ns an objec wi h he nex com-
ponen s:
•ap io i: P obabi ies o each class o he esponse a iable.
• ables: Lis o ables, one o each p edic o a iable. Acco ding he ype o i ,
he in o ma ion ob ained is di e en :
–Quali a i e: he able con ains he p obabili ies o each class o he ou pu
a iable condi ional on he di e en modali ies o he p edic o a iables.
–Quan i a i e: o each class he ou pu a iable, he able con ains he mean
and s anda d de ia ion o he p edic o a iable condi ioned o ha class.
•le els: Ou pu a iable classes.
•call: Sequence o he call o he nai eBayes unc ion.
Finally, once he classi ie has been cons uc ed acco ding o he p e ious unc ions,
p edic ions can be made:
p edic (objec , newda a, ype=class, h eshold=0.001, eps=0)
whe e
•objec : Objec gene a ed by he nai eBayes unc ion.
•newda a: Da a- ame wi h he ins ances o p edic .
54
• ype: I aw is selec ed, he unc ion e u ns he p obabili y o each class o
ou pu a iable ( o each ins ance o p edic ). I class is selec ed, i only indica es
he class wi h maximum p obabili y o each ins ance o p edic .
• h eshold: Value o which he p obabili ies a e eplaced below he alue spec-
i ied in eps.
•eps: The lowe p obabili ies han his alue will be eplaced by h eshold.
Package “klaR”
The e exis s ano he unc ion called Nai eBayes om klaR. The main di e ences
be ween he nai eBayes unc ion (e1071) and he Nai eBayes unc ion (klaR) a e
he ollowing:
•Nai eBayes allows one o speci y p io p obabili ies o classes o he esponse
a iable. In nai eBayes hey a e es ima ed om he lea ning sample.
•Nai eBayes le s one use ke nel unc ion o es ima e he densi y unc ion o con-
inuous a iables. In nai eBayes always i is assumed ha his kind o a iables
ollows a No mal dis ibu ion.
•Nai eBayes equi es lea ning sample gi en as ma ix o da a- ame (i does no
suppo con ingency able).
MIC ou ine
R p o ides a unc ion ha pe o ms he calcula ion o he MIC. The lib a y mi e a
has a unc ion called mine, which e u ns he calcula ion o he MIC o a da a se gi en.
The p incipal a gumen needed is x, a nume ic ec o , ma ix o da a ame (which
is coe ced o ma ix). The es o a gumen s a e op ional and hey a e explained in
he manual p o ided by R. This unc ion e u ns, among o he ou pu s, he Maximal
In o ma ion Coe icien (MIC).
Op imiza ion Rou ines
In his sec ion wo di e en unc ions om wo dis inc packages will be explained.
Fi s , i will be desc ibed a unc ion ha is use ul when uncons ained op imiza ion
p oblems a e ca ied ou . The second one allows one o include cons ain s o he
55
op imiza ion p oblem and, mo eo e , bo h cons ain s as he objec i e unc ion can be
gi en as blackbox unc ions.
Package “d op im”
The lib a y d op im has a unc ion called nmkb. This unc ion is an implemen a ion
o he Nelde -Mead algo i hm o de i a i e- ee op imiza ion. I allows bounds o e
he pa ame e s. The a gumen s o his unc ion a e:
nmkb(pa , n, lowe =-In , uppe =In , con ol = lis (), . . . )
whe e
•pa : A ini ial ec o o pa ame e s. No e ha i mus be be ween lowe and
uppe bounds (i hey exis ).
• n: The nonlinea objec i e unc ion o be op imized.
•lowe and uppe : lowe and uppe bounds on he pa ame e s.
•con ol: a lis wi h he con ol pa ame e s such as ole ance, maximum numbe
o objec i e unc ion e alua ions allowed, . . .
•. . . : auxilia y a gumen s o he objec i e unc ion.
Package “c s”
The lib a y c s has a unc ion called snomad . This las unc ion snomad is an
R in e ace o NOMAD (Nonsmoo h Op imiza ion by Mesh Adap i e Di ec Sea ch)
([1]), an open sou ce so wa e C++ implemen a ion o he Mesh Adap i e Di ec Sea ch
algo i hm designed o cons ained op imiza ion o blackbox unc ions.
This unc ion needs he ollowing inpu a gumen s:
•e al. : he unc ion ha e u ns he alue o he objec i e unc ion and he alue
o he cons ain s.
•n: he numbe o a iables.
•bbin: a ec o ha indica es he ype o he a iables.
•bbou : a ec o ha ixed how he cons ain s ha e o be conside ed du ing he
op imiza ion p ocess.
56
•x0: he ini ial ec o o pa ame e s.
•lb and ub: ec o s wi h lowe and uppe bounds o he con ols, espec i ely.
•. . . : auxilia y a gumen s ha will be needed o he objec i e and cons ain s
unc ions.
No e ha his unc ion has some addi ional a gumen s which a e explained in he
help o R.
57