Semi-Supervised Classification: Mixture models and co-training

Final Deg ee Disse a ion Deg ee in Ma hema ics Semi-Supe ised Classi ica ion: Mix u e models and co- aining Au ho : B uno ´ Al a ez O ega Supe iso : Ja ie C´a camo U iaga June 2025 Con en s In oduc ion 0.1 The case o semi-supe ised classi ica ion . . . . . . . . . . . . . . . . . . Re lexion ii 1 P elimina ies 1 1.1 Supe isedlea ning............................... 1 1.2 Unsupe isedlea ning ............................. 2 1.3 Classi ica ione o ............................... 3 2 Semi-Supe ised Lea ning 5 2.1 Gene a i e mix u e models . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Expec a ion maximiza ion algo i hm . . . . . . . . . . . . . . . . . . . . . 9 2.3 Classi ica ion ule: log-p obabili y a io . . . . . . . . . . . . . . . . . . . 12 2.4 Someca ea s .................................. 13 2.5 Co- aining ................................... 14 2.6 Hypo heses o co- aining . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Compu a ional simula ions 21 3.1 In oduc ion................................... 21 3.2 Fi s se o simula ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3 Second se o simula ions . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4 Example: e ec s o an inco ec model . . . . . . . . . . . . . . . . . . . . 25 3.5 A co- aining simula ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4 Real li e applica ions 31 4.1 Op ical galaxy classi ica ion . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2 Case s udy: ain occupancy da a . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Sel - aining: a inal semi-supe ised app oach . . . . . . . . . . . . . . . . 38 4.4 Conclusion ................................... 40 A R and Py hon code 41 Bibliog aphy 43 iii In oduc ion 0.1 The case o semi-supe ised classi ica ion T adi ionally, classi ica ion has been pe o med by supe ised lea ning algo i hms ained on manually labelled da a o p oduce a good disc imina o o u u e, unseen da a. How- e e , his da a is o en expensi e o ob ain in la ge quan i ies. Wi h he ad en o e e inc easing access o la ge amoun s o da a, mos o i unlabelled, i has become necessa y o le e age his ype o in o ma ion e ec i ely. In his sense, semi-supe ised classi ica ions aims o b idge he gap be ween supe ised and unsupe ised lea ning by de eloping classi ie s ha can make use o bo h he la- belled and unlabelled da a o imp o e classi ica ion pe o mance. Ano he mo i a ion o he unde s anding o semi-supe ised lea ning is i s ela ionship wi h human in elli- gence. Fo example, when in he ea ly yea s o a child pa en s poin o, o ins ance, a small animal a say “dog”, i is he combina ion o his labelled da a and u u e passi e (unlabelled) obse a ions o a dog ha in o m a human being o wha a dog is. This is s udied by cogni i e science, we e classi ica ion algo i hms in o m how humans hink and lea n. In his disse a ion, we will explo e semi-supe ised classi ica ion, s a ing in chap e 1 by e iewing some concep s ele an o his wo k. In chap e 2, we lay down he heo e ical g oundwo k o he wo semi-supe ised me hods s udied: mix u e models and co- aining. In he second hal , chap e s 3, and 4 will pu in o p ac ice hese wo app oaches, wi h some simula ions being pe o med in he o me o es he limi s o heo y. Finally, in chap e 4, we apply his knowledge in h ee di e se case s udies: galaxy classi ica ion, ain occupancy analysis and sen imen ecogni ion. The goal o his disse a ion is o esea ch he e ol ing ield o s a is ical lea ning h ough he pe cep i e o he semi-supe ised lea ning pa adigm. All code de eloped o his wo k is p o ided in he olde CODE, wi h a b ie guide in Appendix A. Re lexion One o he bigges sins ha a young aspi ing ma hema ician can commi is ask abou he ‘use ulness’ o he subjec o s udy. The some imes ha sh esponse om he p o esso s is no wi hou some sense o u h. Some imes knowledge by i sel is aluable enough. Bu soone a he han la e e e y ma hema ician mus eckon wi h one ha sh eali y: ha he beau i ul wo ld o ma hema ics whe e e e y hing makes sense, ull o beau i ul and ewa ding heo ies has a di ec impac on ou socie ies. Indeed, as will became clea h oughou his disse a ion, he opic a hand, semi- supe ised classi ica ion, can build owa ds au oma ion, which will ine i ably lead o mo e decen jobs and economic de elopmen (SDG 8). I also p o ides imp o emen s in e iciency and unde s anding o he complex sys ems ha we ope a e in, aiding in he de elopmen o g een and sus ainable indus ies and in as uc u es (SDGs 9, 11). Classi ica ion is also used daily o medical diagnosis, con ibu ing o he be e men o heal h and wel a e o socie y (SDG 3). Inc easingly, au oma ic classi ica ion has aken a mo e di ec ole in ou li es. In Spain, o ins ance, classi ica ion algo i hms a e used o de e mine he le el o isk ha women ic im o gende iolence ace o p o ide hem wi h he esou ces hey need, wi h he in en o e adica ing his ype o symp om o gende inequali y (SDGs 5, 10). His o y is o en w i en o us because we sh ink om ou bu den as ci izens. We mus ake esponsibili y o ou knowledge and use i o pu sue he goals ou lined by he UN simply because i is he igh hing o do. Fo some imes, he isk o doing no hing becomes he g ea es isk o all. ii Chap e 1 P elimina ies We begin by discussing some concep s ela ed o s a is ical lea ning, in pa icula hose conce ning classi ica ion. De ini ion 1.0.1. Ada a poin o ins ance x= (x1, . . . , xd)∈Rdis he mul i a ia e ep esen a ion o each indi idual in a sample o size n∈N. Ins ances may also be accompanied by a label,y∈ Y, ha ep esen s he class o which xbelongs. In gene al, we may also use he no a ion X o ep esen he se o ins ances. The sample o inpu da a o ou models, whe he i is labelled, unlabelled o pa ially labelled, is o en e e ed o as he aining sample o S. F om o en as amoun s o da a con ained in hese aining samples, one would like o ex ac aluable knowledge. The means and ends o s a is ical lea ning a y g ea ly. Fo now, we discuss he wo main pa adigms in his ield: supe ised and unsupe ised lea ning. 1.1 Supe ised lea ning The main ask o supe ised lea ning is classi ica ion: Gi en a se Yo classes and an ins ance x, ou goal is o classi y his da a poin in one o hose classes (also called popu- la ions), since we belie e he ea u es in xha e an in luence on i s class. In his con ex , all ins ances a e labelled. The pai s (x,y) ep esen he undamen al cha ac e is ic o his pa adigm: i is supe ised in he sense ha labels a e gi en by a “supe iso ”. A ypical example may be he ollowing. Example 1.1.1. All banks ha e a hei disposal he eco ds o pas loans, ha is, he da a co esponding o he clien : sex, age, ma i al s a us, job, income, deposi s, e c. And, on he o he hand, he esul o he ansac ion: whe he he bank made a p o i (i.e. he loan plus in e es we e paid back) o i he clien wen bank up . Gi en a new clien , om whom we know all he abo e-men ioned ea u es, we a e in e es ed in classi- ying i in one o he wo ca ego ies: low- isk, i he ansac ion will likely be success ul, 1 82.1. Gene a i e mix u e models Due o he likelihood unc ion being posi i e and he p ope ies o he loga i hm (con- inuous and mono onic) we always conside he log-likelihood. Recall ha in he con ex o semi-supe ised lea ning, he aining sample is S= {(x1, y1), ..., (xl, yl),xl+1, ..., xl+u}. The e o e, he log-likelihood unc ion is de ined as log L(S|θ) = log l Y i=1 p(xi, yi|θ) l+u Y i=l+1 p(xi|θ)! = l X i=1 log p(xi, yi|θ) + l+u X i=l+1 log p(xi|θ) = l X i=1 log p(yi|θ)p(xi|yi, θ)+ l+u X i=l+1 log p(xi|θ), (2.5) whe e he las equali y is jus i ied by he de ini ion o he condi ional p obabili y. Fo he e ms on he igh , co esponding o he unlabelled da a poin s, he ma ginal p obabili ies p(x|θ) a e conside ed, ha a e, by he o al p obabili y heo em: p(xi|θ) = X y∈Y p(y|θ)p(xi|y, θ) (2.6) Hence, combining (2.5) and (2.6), he log-likelihood unc ion is as ollows. log L(S|θ) = l X i=1 log p(yi|θ)p(xi|yi, θ) + l+u X i=l+1 log X y∈Y p(y|θ)p(xi|y, θ). (2.7) I is easy o see ha he log-likelihood ha we a e ying o maximize is only di - e en om ha o he supe ised lea ning o he e ms in he second sum. This is a c i ical di e ence, since unlike in such a lea ning pa adigm, he op imiza ion p oblem in semi-supe ised lea ning is no necessa ily a con ex p oblem, which makes i mo e challenging o sol e. In addi ion, he solu ion o he equa ion ∇log L(S|θ) = 0canno , in gene al, be explici ly calcula ed, and he e o e a nume ical me hod is equi ed. In his con ex , he s anda d me hod o sol ing o he MLE p oblem is he Expec a ion Maximiza ion algo i hm (EM), which inds a local maximum o he objec i e unc ion p esen ed in (2.7). Chap e 2. Semi-Supe ised Lea ning 9 2.2 Expec a ion maximiza ion algo i hm De ini ion 2.2.1. Gi en a aining sample S={(x1, y1), ..., (xl, yl),xl+1, ..., xl+u}, we call hidden a iables o he unknown labels o he unlabelled ins ances, H={yl+1, ..., yl+u}. The expec a ion maximiza ion algo i hm is an i e a i e me hod which inds a local op i- mum o θ, he model pa ame e s. Gi en some ini ial pa ame e alues, ˆ θ(0), he me hod epea s he ollowing wo s eps un il a ia ion in he log L(S|θ) is below a ce ain ε. (i) Expec a ion s ep: The expec ed alue o he hidden a iables in His calcu- la ed. This can be hough o as “so assignmen s” o he labels o ins ances xl+1, ..., xl+u. Fo mally, we compu e p( )(H), which by de ini ion is p(H|S,ˆ θ( )). In Algo i hm 1 , s ep 2a, a o mula o he compu a ion o hese p obabili ies is p esen ed o he case o a wo componen GMM model. (ii) Maximiza ion s ep: These “so labels”, ˆ H( ), a e used o upda e he pa ame e s, by calcula ing he MLE o {(x1, y1), ..., (xl, yl),(xl+1,ˆy( ) l+1), ..., (xl+u,ˆy( ) l+u)}. In o he wo ds, ˆ θ( +1) is calcula ed, such ha i maximizes Q(θ|ˆ θ( )) = Elog pS,ˆ H( )|θ. Depending on he ini ial condi ions, ˆ θ(0), he local maximum ha EM inds may a y. Usually, ˆ θ(0) is chosen as he MLE o he labelled da a called, Slabelled. Equi alen ly, ˆ θ(0) = a g max θlog L(Slabelled|θ) = a g max θ l X i=1 log p(yi|θ)p(xi|yi, θ).(2.8) We now gi e a lemma necessa y o p o e an impo an esul o he EM algo i hm. Lemma 2.2.1. (Gibb’s inequali y) Le {p(x)}x∈X and {q(x)}x∈X be wo disc e e p ob- abili y dis ibu ions. Then, −X x∈X p(x) log q(x)≥ − X x∈X p(x) log p(x).(2.9) Gibb’s inequali y is o en s a ed his way because o i s ela ion o he concep o en opy o quan i y o in o ma ion o a andom a iable. See [3, pp. 287 - 289] o mo e de ails and he co esponding p oo . Theo em 2.2.2. The expec a ion maximiza ion me hod imp o es he log-likelihood, log L(S|θ), in e e y successi e i e a ion. P oo . By de ini ion o condi ional p obabili y, p(H | S, θ) = p(S,H | θ) p(S | θ)=⇒log p(S | θ) = log p(S,H | θ)−log p(H | S, θ). 10 2.2. Expec a ion maximiza ion algo i hm We conside he expec ed alue o all possible Husing he es ima ion o he pa ame e s o he cu en i e a ion. This is done h ough mul iplica ion by p(H | S,ˆ θ( )) on bo h sides, and aking he sum, log p(S | θ) = X H p(H | S,ˆ θ( )) log p(S,H | θ) −X H p(H | S,ˆ θ( )) log p(H | S, θ). No e ha he le -hand side s ays he same since i is a cons an . In he igh -hand side o he equa ion, he i s sum is wha we ha e de ined as Q(θ|ˆ θ( )) ( he expec ed alue o he log-likelihood as a unc ion o θ, gi en he cu en es ima e, ˆ θ( )) and minus he second sum, which we deno e as H(θ|ˆ θ( )). In his way, we ha e he ollowing iden i y, log p(S | θ) = Q(θ|ˆ θ( )) + H(θ|ˆ θ( )).(2.10) Since equali y (2.10) is ue o any alue o θ, we can make θ=ˆ θ( +1) and θ=ˆ θ( ) o ob ain, espec i ely, log p(S | ˆ θ( +1)) = Q(ˆ θ( +1) |ˆ θ( )) + H(ˆ θ( +1) |ˆ θ( )) and (2.11) log p(S | ˆ θ( )) = Q(ˆ θ( )|ˆ θ( )) + H(ˆ θ( )|ˆ θ( )).(2.12) Sub ac ing equa ion (2.12) o (2.11) yields log p(S | ˆ θ( +1))−log p(S | ˆ θ( )) = Q(ˆ θ( +1) |ˆ θ( ))−Q(ˆ θ( )|ˆ θ( )) +H(ˆ θ( +1) |ˆ θ( ))−H(ˆ θ( )|ˆ θ( )).(2.13) Using Lemma 2.2.1 (Gibb’s inequali y) o p obabili y dis ibu ions P={p(H | S,ˆ θ( ))}H and Q={p(H | S,ˆ θ( +1))}Hwe ha e ha H(ˆ θ( +1) |ˆ θ( ))≥H(ˆ θ( )|ˆ θ( )) and hus, log p(S | ˆ θ( +1))−log p(S | ˆ θ( ))≥Q(ˆ θ( +1) |ˆ θ( ))−Q(ˆ θ( )|ˆ θ( )).(2.14) Finally, since in e e y i e a ion in he maximiza ion s ep we calcula e ˆ θ( +1) o imp o e Q(ˆ θ( +1) |ˆ θ( )), and knowing ha log p(S | θ) is he log L(S|θ), we ob ain he desi ed esul . In he ollowing page, we p esen he o mula ion o he EM algo i hm o a wo- componen Gaussian mix u e model ( om now on GMM), as in Example 2.1.1. We aim o es ima e he p io p obabili ies, π0,π1and he pa ame e s o he wo no mal popula ions, (µ0,Σ0) and (µ1,Σ1). All in all, ou model pa ame e s a e θ={πj, µj,Σj}j∈{0,1}. Chap e 2. Semi-Supe ised Lea ning 11 Algo i hm 1. EM o GMM: Inpu : Sample S={(x1, y1), ..., (xl, yl),xl+1, ..., xl+u}, and ole ance ε > 0. (i) Ini ializa ion: Make = 0 and ˆ θ(0) ={ˆπ(0) j,ˆµ(0) j,ˆ Σ(0) j}j∈{0,1}, he MLE o he labelled da a. Fo ins ance, ˆπj=|{xi∈Slabelled :yi=j}| |Slabelled|. (ii) I e a e he ollowing s eps un il con e gence o he log L(S|θ) is achie ed. Equi a- len ly, he p ocess is s opped i log L(S | ˆ θ( +1))−log L(S | ˆ θ( ))≤ε . (a) Expec a ion s ep: Fo all he unlabelled ins ances, i∈ {l+ 1, . . . , l +u}, calcula e using Bayes’ ule, γij := p(yj|xi,ˆ θ( )) = ˆπ( ) jN(xi; ˆµ( ) j,ˆ Σ( ) j) P1 k=0 ˆπ( ) kN(xi; ˆµ( ) k,ˆ Σ( ) k), j = 0,1. These alues can be hough o as ac ional labels es ima ed o he unla- belled da a poin s. On he o he hand, o he labelled ins ances, conside he ue assignmen s, ha is, o i∈ {1, . . . , l}, γij =(1,i yi=j , 0,o he wise . (b) Maximiza ion s ep: Calcula e ˆ θ( +1), o j∈ {0,1}as he MLE o he aining sample Swi h he ac ional labels γ. lj= l+u X i=1 γij , ˆµ( +1) j=1 lj l+u X i=1 γijxi, ˆ Σ( +1) j=1 lj l+u X i=1 γij(xi−ˆµ( +1) j) (xi−ˆµ( +1) j), ˆπ( +1) j=lj l+u. (iii) Upda e := + 1. Re u n o s ep (ii). Ou pu : MLE {ˆπj,ˆµj,ˆ Σj}j∈{0,1}. 12 2.3. Classi ica ion ule: log-p obabili y a io The EM algo i hm is an example, in he ield o compu e science, o a sel - aining algo- i hm, o an algo i hm ha eaches i sel . This is because o he unlabelled ins ances, an es ima e o hei labels is calcula ed, which a e hen used o augmen he es ima ion o he model pa ame e s, using MLE as i he whole sample was labelled. The p oblem o he EM algo i hm no inding he global op imum o he log-likelihood bu a he a local maximum can be deal wi h in a numbe o di e en ways. One ap- p oach is wha is o en e e ed o as a andom s a , whe e he ini ial alues o EM, ˆ θ(0), a e andomly chosen. This p ocess is epea ed, and only he bigges log-likelihood achie ed is conside ed. I is clea ha his is jus a heu is ic app oach and does no gua an ee he op imal solu ion. Ano he app oach would be u ilizing nume ical me h- ods o sol ing uncons ained op imiza ion p oblems, such as g adien descen , which again does no gua an ee he global op imum i he solu ion space is no con ex and would also equi e he andom s a me hodology. 2.3 Classi ica ion ule: log-p obabili y a io We ecall ha ou o iginal objec i e o gene a i e mix u e models was classi ica ion. Since any classi ica ion ask whe e he e a e mo e han wo classes can be educed o he p oblem o bina y classi ica ion, le us assume ha Y={0,1}, which we e e o as he se o he 0 and 1 class. In o de o classi y each x∈ S o any new ins ance x∈Rd, so as o assess he ac- cu acy o classi ica ion, by he o al p obabili y heo em we ha e, p(x|y= 0)p(y= 0) + p(x|y= 1)p(y= 1) = p(x).(2.15) This exp ession allows us o es ima e he p obabili y o an ins ance coming om he 1 class: p(y= 1|x) = 1 p(x|y=0)p(y=0) p(x|y=1)p(y=1) + 1 .(2.16) No e ha ha ing es ima ed he model pa ame e s using he expec a ion maximiza ion me hod, bo h he p io p obabili ies and he class condi ional dis ibu ions a e known. I is also in e es ing o men ion ha o mula (2.16) is simila o ha o he logis ic eg ession o logi model. This is why, unde a GMM model, whe e p(x|y= 0) = N(x|µ0,Σ0) and p(x|y= 1) = N(x|µ1,Σ1), we can use o mula (2.3) o expand (2.16) and achie e he ollowing exp ession o he log-p obabili y a io: log p(y= 1|x) p(y= 0|x)!=1 2(x−µ0) Σ−1 0(x−µ0)−(x−µ1) Σ−1 1(x−µ1) +1 2log |Σ0| − log |Σ1|+log p(y= 1) −log p(y= 0). (2.17) Chap e 2. Semi-Supe ised Lea ning 13 The log p obabili y a io allows us o gi e he ollowing classi ica ion ule: ˆy= 1 ⇐⇒ log p(y= 1|x) p(y= 0|x)!>0.(2.18) Fu he mo e, exp ession (2.17) gi es a measu e o he con idence o classi ica ion: he u he he a io is om 0, he mo e likely is o he classi ica ion o be co ec . We can jus i y his by conside ing he h ee main componen s o he log p obabili y a io: (i) (x−µ0) Σ−1 0(x−µ0)−(x−µ1) Σ−1 1(x−µ1)is he di e ence o he squa es o he Mahalanobis dis ance be ween he ins ance and each o he means. The g ea e he absolu e alue o his di e ence, he close he ins ance is o one o he dis ibu ions compa ed o he o he . (ii) (log |Σ0| − log |Σ1|) he di e ence o he gene alized log- a iances. (iii) log p(y= 1) −log p(y= 0) he di e ence be ween he p io p obabili ies. 2.4 Some ca ea s We now ha e a clea semi-supe ised me hod ha uses he unlabelled da a o imp o e he accu acy o classi ica ion. Ne e heless, one mus be cau ious abou he co ec ness o he model o he unde lying hypo hesis: ha he da a is ac ually gene a ed by he mix u e model ha is being conside ed. In o he wo ds, should he numbe o com- ponen s (which may no necessa ily be |Y|), he p io p obabili ies, o he condi ional p obabili y dis ibu ions p(x|y) be inco ec , he accu acy o he p edic o migh be less han i only labelled da a was used in a adi ional supe ised lea ning con ex . In chap- e 3, some simula ions will be p esen ed o illus a e his poin mo e p ecisely. On he o he hand, domain knowledge is use ul in o de o conside a gene a i e model. Fo example, image analysis o medical ials a e ields whe e a simple s a is ical analysis shows popula ions ypically ollow a Gaussian dis ibu ion. In hese con ex s, a Gaussian Mix u e Model wi h he app op ia e numbe o componen s would be adequa e. 14 2.5. Co- aining 2.5 Co- aining Co- aining [11] is ano he impo an semi-supe ised classi ica ion me hod. I is spe- cially sui able o wha is b oadly e e ed o as na u al language p ocessing. In pa - icula , we will concen a e on named en i y classi ica ion, which is a ask ha in ol es classi ying a p ope name in o one o mul iple classes depending on i s meaning. Conside he ollowing illus a i e example. Example 2.5.1. Suppose we a e in e es ed in classi ying Wikipedia a icles in o one o he wo ollowing ca ego ies: humans o places, he i s ones being biog aphical accoun s o some pe son, and he la e being abou geog aphical spaces. Conside ha we a e gi en as ins ances x= (x(1),x(2)), whe e x(1) is he i le o he a icle and x(2) is an exce p o he abs ac . One aining sample Scould be, Ins ance x(1) x(2) y 1 Joseph Fou ie ...ma hema ician and physicis ... pe son 2 Co sica ...island in he Medi e anean Sea... place 3 Leonha d Eule ...ma hema ician, ..., as onome ... ??? 4 Madagasca ...is an island coun y... ??? 5 Leona do da Vinci ...as onome and a chi ec ... ??? ... ... ... ... As always, anno a ed da a is di icul o ob ain because i equi es manual labou (in his case we only ha e wo labelled ins ances) bu we ha e plen y o unlabelled ins ances a ou disposal. A simple co- aining classi ie ha uses bo h he name o he a icle (x(1)), as well as he con ex (gi en by x(2)) o lea n name en i y classi ica ion would p oceed as ollows: (i) F om ins ance 1 we lea n ha “ma hema ician” and “physicis ” appea in he con ex o he label “pe son”. The same idea applies o Co sica: he classi ie lea ns ha “island” co esponds o a place. (ii) Knowing his, we a e able o classi y Madagasca as a place, as i has “island” in i s con ex . (iii) Simila ly, we can assign he class “pe son” o Leona d Eule , and his in u n allows us o lea n “as onome ” o be associa ed wi h “pe son”. (i ) Finally, his would enable he classi ica ion o Leona do da Vinci as a “pe son” e en hough nei he he name no he con ex we e p esen in he anno a ed da a. As in he example, we conside each ins ance x o be desc ibed by wo ea u e se s, also called iews, (x(1),x(2))∈Rd1×Rd2. This is o en he case o eal-wo ld da a. Take, o ins ance, con en mode a ion in a social media pla o m like YouTube, whe e each ideo is desc ibed by i s me a-da a ( i le, desc ip ion, e c.) and he con en o he Chap e 2. Semi-Supe ised Lea ning 15 ideo i sel . YouTube’s algo i hm may decide whe he a ideo is sui able o ecommen- da ion based on hese wo iews. Fo mally, a co- aining algo i hm is an ensemble me hod ha uses wo dis inc clas- si ie s, (1) and (2), which a e only ained wi h he labelled ins ances, aking in o accoun solely he iews x(1) and x(2), espec i ely. The mos con iden p edic ions o each classi ie a e added o he labelled da a o he o he . In his way, bo h classi ie s each one ano he . Algo i hm 2. Co-T aining Inpu da a: S={(x(1) 1, y1),(x(2) 1, y1),...,(x(1) l, yl),(x(2) l, yl),x(1) l+1,x(2) l+1,...,x(1) l+u,x(2) l+u}, and k∈N, wi h k≤l+u, he lea ning speed. (i) Conside he aining da a se s o classi ie s (1) and (2) o be, espec i ely, L1={(x(1) 1, y1),...,(x(1) l, yl)},L2={(x(2) 1, y1),...,(x(2) l, yl)}. (ii) While S (L1∪L2)=∅pe o m he ollowing s eps: (a) T ain (1) om L1and (2) om L2. (b) Use bo h (1) and (2) o classi y ins ances om S (L1∪L2). (c) Fo each classi ica ion o he p e ious ins ances made by (1) and (2), a con idence alue is assigned. Conside {x(1) ij,ˆy(1) ij}k j=1 and {x(2) i′ j,ˆy(2) i′ j}k j=1 o be he k-mos con iden p edic ions o (1) and (2), espec i ely. Add hese o he aining sample o he o he classi ie : •L1=L1∪ {(x(1) i′ j,ˆy(2) i′ j)}k j=1 . •L2=L2∪ {(x(2) ij,ˆy(1) ij)}k j=1 . Classi ie s (1) and (2) only pay a en ion o hei co esponding iews, bu he ain- ing ins ances may be gi en by he o he classi ie s, hence he no a ion ˆy(2) i′ jand ˆy(1) ij, espec i ely, in he upda ed aining samples. No ice also ha we a e no in e es ed in he speci ic na u e o each supe ised lea ne , bu on he way in which hey can be combined o imp o e classi ica ion in he con ex o semi-supe ised lea ning, when only a ew anno a ed ins ances wi h wo iews a e a ailable. The only equi emen o (1) and (2) is o be able o assign a con idence o accu acy o hei p edic ions. (1) and (2) a e o en e e ed o as iew-1 and iew-2 classi ie s, espec i ely. Co- aining is one o many semi-supe ised lea ning me hods ha u ilizes he “dis- ag eemen s” be ween he wo classi ie s ained on a smalle , ully labelled da a se , and e- ains hem un il hey ag ee on a la ge sample, using he unlabelled da a. 16 2.6. Hypo heses o co- aining 2.6 Hypo heses o co- aining The i s necessa y condi ion in o de o apply co- aining is o he ins ances o ha e wo iews. Besides he con ex s in which his happens na u ally, ins ances can be a bi a ily spli in o wo iews. Ne e heless, o co- aining o be success ul in any o hese cases, wo main hypo heses a e usually conside ed: iews x(1) and x(2) mus be su icien and edundan ( o he classi ica ion ask) and condi ionally independen [12]. The su icien and edundan hypo hesis asks o bo h iews o be su icien ly in o - ma i e. In o he wo ds, ha a good classi ie can be ained solely on each x(1) and x(2). On he o he hand, iews mus be condi ionally independen . Tha is, p(x(1) |y, x(2)) = p(x(1) |y), p(x(2) |y, x(1)) = p(x(2) |y).(2.19) This means ha gi en he label y, knowledge o one o he iews does no a ec he p obabili y o he o he . To illus a e his hypo hesis, le ’s ecall ou example, in which iew x(1) was he i le o a Wikipedia a icle, and x(2) a agmen o i s abs ac . Fix he label y= “pe son” and conside o example he con ex x(2) = “ma hema ician and physicis ”. The condi ionally independen hypo hesis implies ha his con ex does no bene i any pa icula i le o name, gi en he ue label y= “pe son”. This hypo hesis, p oposed by Blum and Mi chell in 1998 (see [5, pp. 92 - 100]), is gene ally o e ly s ic and usually does no hold, e en wi h la ge da a se s. This is ob ious in ou example, since x(2) = “...F ench empe o ...” hea ily in luences he a icle name. None heless, i is in ui i e o unde s and why his hypo hesis is conside ed, e en i i is no always ue. I condi ional independence did no gene ally hold and some x(2) we e lea n by (2) o be associa ed wi h a pa icula class, since his classi ie eaches (1) by adding he co esponding iew o i s aining da a, we would isk adding less and less in o ma i e ins ances o (1), as a esul o hese being o e ly “simila ” o each o he and hence de ea ing he pu pose o imp o ing classi ica ion. The e o e, a elaxa ion o his necessa y hypo hesis is p oposed by Abney in [1], in which an uppe bound o he classi ica ion e o is gi en in e ms o he a e o dis- ag eemen o he wo lea ne s, as we ha e men ioned be o e. To unde s and his, we need o in oduce he ollowing concep s: Le X1be he space whe e x1belongs. Conside H1 o be all possible classi ie s om X1 o Y. Analogously, o he second iew we ha e X2and H2. De ini ion 2.6.1. Gi en y∈ Y ={0,1}, (1) ∈ H1and (2) ∈ H2, we say ha classi ie s (1) and (2) a e condi ionally independen i o all u, ∈ Y : p( (1) =u| (2) = , y) = p( (1) =u|y), p( (2) =u| (1) = , y) = p( (2) =u|y).(2.20) Chap e 2. Semi-Supe ised Lea ning 17 Remembe ha (1) ∈ H1is a iew-1 only classi ie , i.e., (1) :X1−→ Y, analogously wi h (2). Bo h concep s o condi ional independence a e ela ed o each o he by he ollowing esul . P oposi ion 2.6.1. I iews x(1),x(2) a e condi ionally independen , hen (1) and (2) a e condi ionally independen . P oo . By de ini ion o condi ional independence o iews x(1) and x(2), gi en y, u, ∈ Y, p( (1) =u| (2) = , y) = p{x(1) : (1)(x(1)) = u}|{x(2) : (2)(x(2)) = }, y =p{x(1) : (1)(x(1)) = u} | y =p( (1) =u|y). As we ha e obse ed, his hypo hesis is gene ally un easonable, hough use ul. We now in oduce a measu e o how ou classi ie s de ia e om he condi ional independence hypo hesis. De ini ion 2.6.2. The condi ional dependence o (1) and (2) gi en yis dy=1 2X u, ∈{0,1}p( (2) = |y, (1) =u)−p( (2) = |y).(2.21) The e o e i (1) and (2) a e condi ionally independen hen dy= 0. This no ion will allow us o elax he condi ional independence hypo hesis by allowing dy o be bounded. De ini ion 2.6.3. Fo y∈ Y ={0,1}, le p1= minu∈Y{p( (1) =u|y)},p2= minu∈Y {p( (2) =u|y)}and q1= 1 −p1. Then, (1) and (2) sa is y weak dependence i : dy≤p2 q1−p1 2p1q1 .(2.22) No e ha i p1= 1/2, hen dy= 0 and we eco e condi ional independence. Indeed, p1 and p2canno be g ea e han 1/2 by de ini ion. In pa icula , p1se es as a measu e o how much condi ional dependence we can ha e and is o en e e ed o as he mino i y p obabili y o classi ie (1): he lowe p1is, he g ea e dycan be. Le he mino i y alue o a classi ie be he label whe e he mino i y p obabili y is achie ed. Ano he use ul concep ha will become impo an la e on is he a e o disag ee- men o classi ie s (1) and (2), which is simply he p( (1) = (2)) = p({(x(1),x(2))| (1)(x(1))= (2)(x(2))}). De ini ion 2.6.4. P edic o s (1) and (2) a e said o be non- i ial i minu∈Y {p( (1) = u)}> p( (1) = (2)). 24 3.3. Second se o simula ions These simula ions unde sco e he impo ance o semi-supe ised lea ning echniques, in pa icula he mix u e model app oach o enhance classi ica ion pe o mance when plen y o unlabelled da a is a ailable and a basic knowledge o he unde lying assump- ions can be le e aged e ec i ely. 3.3 Second se o simula ions We now es he obus ness o he semi-supe ised me hodology when he unde lying mix u e model assump ions a e inco ec o analyse how i s pe o mance decays and compa es o he supe ised lea ning echniques. Figu e 3.3 Conside he da a ep esen ed in he his og am in Figu e 3.3. I is a mix u e model o wo dis inc dis ibu ions wi h weigh 1/2: he le -mos componen is a -s uden wi h 10 deg ees o eedom cen ed a µ0=−2, while he second componen is an skewed no mal dis ibu ion o pa ame e s ξ= 2, ω= 1.5 and α= 2, meaning i is a sligh ly asymme ic - con olled by pa ame e α- no mal dis ibu ion cen ed a ound 3. E en hough he model assump ions o a wo componen Gaussian mix u e model a e no co ec , we would be o gi en o ying o i such a model o he da a. Indeed, i we es ima e he model pa ame e s using he EM algo i hm o a wo componen Gaussian mix u e model and use hem o pe o m he s anda d Kolmogo o -Smi no no mali y es on ou da a, we ob ain a p- alue o 0.9471. Thus, we a e unable o dis ega d no - mali y. As in he p e ious sec ion, in o de o assess he beha iou o he semi-supe ised ap- p oach unde ailu e o he model hypo heses, we pe o m 100 simula ions whe e he da a was andomly gene a ed om he mix u e model p esen ed abo e. One hund ed poin s o each class we e gene a ed in each ial, o a o al sample size o n= 200, along wi h wo andomly selec ed labelled ins ances om each componen . Chap e 3. Compu a ional simula ions 25 The esul s we e as ollows. Using LDA ained on he ou labelled ins ances ( he es is conside ed unlabelled), he a e age decision bounda y o he 100 ials was 0.45, wi h a s anda d de ia ion o 0.66, indica ing, as commen ed be o e, ha classi ica ion wi h LDA is highly dependen on he speci ic labelled ins ances chosen o aining. None heless, he a e age classi ica ion e o was 4.68%†. SVM yielded simila esul s when ained on he e y same labelled poin s, wi h he mean bounda y a −0.051. This me hod was also mo e s able, o he a e age de ia ion o he decision bounda y was jus 0.12. Con a y o he p e ious simula ions, classi ica ion e o wi h SVM was sligh ly wo se a 5.68%. Conside ing now he a o emen ioned semi-supe ised app oach based on he EM al- go i hm, a wo componen GMM has been assumed o ha e gene a ed he da a. Fu - he mo e, ini ializa ion in all ials o EM was conside ed such ha he means we e chosen a −1 and 1, wi h a s anda d de ia ion o 1 o bo h componen s, as well as equal p io p obabili ies. The summa y o he a e age esul s ound by EM is as ollows. bµ0bµ1bσ0bσ1bπ0bπ1 −2.309 2.852 1.332 1.332 0.452 0.548 Table 3.2: A e age es ima ed pa ame e s o he GMM o e all simula ions. The al eady discussed clus e hen label s a egy p o ided an a e age classi ica ion a e o 4.55%, ma ginally be e han he LDA app oach. This indica es ha e en i he unde lying hypo heses a e no ue, he p obabili y dis ibu ions in hese simula ions a e close enough o no mali y ha e en unde he ailu e o he model co ec ness hy- po hesis, he me hod s ill boos ed classi ica ion pe o mance. In conclusion, lack o he model co ec ness hypo hesis does no necessa ily hu , in gene al, he model’s pe o mance. Ne e heless, le us see an example whe e his does indeed happen. 3.4 Example: e ec s o an inco ec model As men ioned, inco ec assump ions in he gene a i e model need no be ca as ophic, bu can ac ually, unde ce ain ci cums ances, hu i s pe o mance when compa ed o adi ional supe ised classi ica ion me hods. Le us unde s and his phenomenon wi h he ollowing example. Below, in Figu e 3.4a, a sample o 200 unlabelled da a poin s is ep esen ed. Clea ly, wo clus e s in he o m o wo ellipses a e isible. To he igh , in Figu e 3.4b, he ue class o each poin is displayed wi h blue do s o he ze o class and wi h ed iangles o he one class. †E o a es in his sec ion we e calcula ed labelling he da a based on he es ima ed decision bounda y wi h each me hod and compa ing hem o he ue labels. 26 3.4. Example: e ec s o an inco ec model (a) Sample da a. (b) Two classes in ou clus e s. Figu e 3.4 Knowledge o he ue unde lying model allows us o iden i y ou dis inc bi-dimensional no mal dis ibu ions ha ha e gene a ed he poin s om he wo popula ions ( ed and blue) in he wo clus e s. Wi hou his p io insigh in o he model, hough, we could easonably assume ha he da a comes om a wo-componen no mal mix u e model, when in eali y he e a e ou no mal dis ibu ions ha ha e gene a ed i , as in Fi- gu e 3.4b. In spi e o he model no being co ec , i is he mos easonable because i is he one wi h he highes log-likelihood, e en when compa ed o he wo componen model co esponding o he ue popula ions. This is shown in Figu e 3.5, whe e he ellipses in he le diag am ep esen he assumed model, which indeed has a g ea e log-likelihood compa ed o ha o he ac ual model ep esen ed in he igh -hand side diag am. (a) Assumed model. log-likelihood: -849.0514. (b) Co ec model. log-likelihood: -1029.025. Figu e 3.5 Chap e 3. Compu a ional simula ions 27 The assumed model, a wo componen GMM, was es ima ed using a pu ely unsupe ised EM algo i hm. This model i s he unlabelled da a easonably well, wi h an es ima ed log-likelihood o −849.051. The wo ellipses isible in Figu e 3.5a a e cen ed a he es ima ed means o each o he popula ions and co espond o he calcula ed co a iance ma ices. On he con a y, he co ec model in Figu e 3.5b ac ually adap s wo se o he da a, and hence he log-likelihood is lowe a −1029.025. In ac uali y, howe e , depending on he labelled da a p o ided o ou classi ie ( he a- ssumed model), he es ima ed decision bounda y will app oxima ely be y=−x, which would p oduce an app oxima e classi ica ion e o o 0.5, making his classi ie i ually useless. This phenomenon, whe e he semi-supe ised app oach would app oxima ely lea n such a linea classi ie is ep esen ed in Figu e 3.6a. To he igh , he ue decision bounda y based on he co ec model assump ions is displayed. (a) Decision bounda y a y=−x o an app oxima e e o o 0.5. (b) Co ec decision bounda y a y=x. This is he Bayes classi ie . Figu e 3.6 I is he e o e e iden h ough his example ha a ypical supe ised classi ie , like an SVM, e en when u ilizing he same labelled ins ances, would ha e a simila o e en be e classi ica ion pe o mance. This is especially ue i a ew mo e labelled ins ances we e p o ided. In conclusion, he semi-supe ised app oach does no gua an ee g ea e classi ica ion accu acy han i s supe ised pee s, bu equi es domain knowledge o a basic unde s anding o he gene a i e model o boos pe o mance. Rema k 3.4.1. When domain knowledge is no a ailable o he gene a i e model is unclea , i is use ul o employ a semi-supe ised non-pa ame ic app oach o es ima e he densi y unc ions o he di e en sub-popula ions. 28 3.5. A co- aining simula ion Indeed, i we use he unc ion m npEM om he R package mix ools, we can use his new non-pa ame ic EM algo i hm on ou o iginal unlabelled da a se o be e iden i y he ue subpopula ions. This ke nel based EM algo i hm o iginally sea ches o 6 sub- popula ions in he da a. The esul s a e shown below, in Figu e 3.7. Figu e 3.7: Fou mix u e componen s a e ound by he non-pa ame ic EM. When he p ocess is comple ed, he algo i hm is able o assign e e y poin o one o ou sub-popula ions, which clea ly o e lap wi h he ue classes in Figu e 3.4b. A ew cen ally loca ed labelled da a poin s in each o he mix u e componen s would hen allow us o classi y wi h an accu acy close o ha o he Bayes’ classi ie . 3.5 A co- aining simula ion Le us now analyse how o enhance classi ica ion pe o mance using he co- aining amewo k s udied in Sec ion 2.5. In his con ex , ins ances ha e wo iews o se s o ea u es. Conside supe ised classi ie s h(1) and h(2) as iew-1 and iew-2 only, which we will e e o as ou “base” classi ie s, since hey a e only ained on he o iginal labelled samples, say L1={(x(1) 1, y1),...,(x(1) l, yl)},L2={(x(2) 1, y1),...,(x(2) l, yl)}, espec i ely. Ou goal is o make use o all he ini ially unlabelled da a in ou sample, {x(1) l+1,x(2) l+1,...,x(1) l+u,x(2) l+u}, o co- ain wo new be e classi ie s, (1) and (2). In ou simula ion, 1000 no mally dis ibu ed da a poin s wi h wo iews we e gene a- ed, 500 o each o he wo classes. Fo he 0 class, he poin s om he i s iew ollow he andom a iable Xy=0, iew-1 ∼ N (−1.5,1), while hose o he second iew a e d awn om Xy=0, iew-2 ∼ N (−1,1.2). Simila ly, da a om he 1 class is such ha Xy=1, iew-1 ∼ N(1.5,1) and Xy=1, iew-2 ∼ N (1,1.2). All in all, o he 0 class, he bi- dimensional poin s a e cen ed a (−1.5,−1), while a (1.5,1) o he 1 class. Chap e 3. Compu a ional simula ions 29 Figu e 3.8: Gene a ed syn he ic da a. Figu e 3.8 abo e displays he 1000 da a poin s wi h hei wo uni a ia e iews ac ing as ea u es. F om his plo , i is clea ha he wo classes a e able o be sepa a ed by a linea classi ie . Fo each o he wo iews, he densi y unc ion o he gene a ed da a is shown in he ma gins, sepa a ed by class, which co espond o he no mal dis ibu ions conside ed ea lie . In o de o be e assess classi ica ion pe o mance, 20% o ha da a was se apa as he es sample. O he emaining, 5% was conside ed labelled, wi h he o he 95% o aining ins ances being unlabelled. Bo h he “base” classi ie s h(1),h(2) and h ough co- aining (1), (2) we e chosen o use he usual LDA supe ised classi ica ion ech- nique. Indeed, ini ially, h(1) = (1) and h(2) = (2). Co- aining was pe o med wi h a lea ning speed k= 10, as desc ibed in Algo i hm 2, whe e he k-mos con iden p edic ions o each classi ie a e added (wi h hei p ojec ed labels) o he aining sample o he o he . Then, each classi ie is e- ained. The algo i hm s ops when he e is no longe any unlabelled ins ances, o in addi ion, i classi ica ion accu acy o (1) and (2) deg ades o does no imp o e o 3 i e a ions. The inal esul s, ob ained a e only 4 i e a ions, a e summa ized in Table 3.3. Classi ie Accu acy Base iew-1 0.885 Base iew-2 0.775 Co- ained iew-1 0.890 Co- ained iew-2 0.800 Table 3.3: Final esul s o he ou classi ie s. 30 3.5. A co- aining simula ion The ollowing diag am, Figu e 3.9, shows he p og ess in accu acy o he classi ie s h oughou he ou i e a ions. The do ed lines co espond o he base classi ie s, which emain unchanged, while he co- ained classi ie s a e ep esen ed by solid lines. Figu e 3.9: Accu acy o he ou classi ie s om i e a ion 0, whe e h(1) = (1) and h(2) = (2), o i e a ion ou . Base classi ie s h(1) and h(2) emain unchanged because hey a e ained on he o iginal labelled sample. Bo h iew-1 and iew-2 classi ie s ained wi h co- aining p og essi ely imp o e hei accu acy, by 0.5% and 2.5%, espec i ely. The unimp essi e imp o emen s in classi i- ca ion accu acy a e due o he ac ha he wo iews, by hemsel es, a e in o ma i e enough o ain a good base classi ie . In addi ion, nei he o he wo iews is mo e in o ma i e han he o he and hey a e no condi ionally independen , so co- aining is only able o mo e pe o mance sligh ly close o he heo e ical maximum. Ne e heless, his demons a es he powe o co- aining o boos classi ica ion pe o mance unde he igh assump ions and amewo k. Chap e 4 Real li e applica ions 4.1 Op ical galaxy classi ica ion G a i y bounds galaxies oge he o o m wha as onome s call galaxy clus e s. The numbe o galaxies in hese clus e s can ange om hund eds o housands. Ne e heless, apa om clus e ing galaxies in he physical space, hey can also be clus e ed in he colou space based on hei obse ed equency in he elec omagne ic spec um. This is ela ed o he concep o “ edshi ”, a g a i a ional phenomenon p edic ed by he heo y o gene al ela i i y ha s a es ha pho ons emi ed om he cen e o galaxies lose ene gy and he e o e become mo e ed in he spec um. This op ical clus e ing is o in e es o ou disse a ion, since he colou o a galaxy can be used o unde s and i s na u e. Mos galaxies can be classi ied in one o he wo ollowing ca ego ies: ed sequence galaxies (RS) ha e gene ally low s a o ma ion ac i i y and a e usually ellip ical, and blue cloud (BC) galaxies ha a e o en o spi al o m and a e p oducing new s a s. Galaxies ha do no belong o ei he o he wo a o emen ioned ca ego ies - like ou s, he Milky Way - li e in he “g een alley”. S a o ma ion ac i i y is measu ed ia he speci ic s a o ma ion a e o sSFR. Unde low edshi (meaning o galaxies ha a e ela i ely close), he colou o a galaxy is a good p oxy o he sSFR. As discussed, RS galaxies usually ha e low sSFR, while BC galaxies gene ally ha e high sSFR. Wha is in e es ing is ha he dis ibu ion o he speci ic s a o ma ion a e ac oss galaxies is o bi-modal na u e, which sugges s he use o a wo componen Gaussian mix u e model o disc imina e be ween RS and BC galaxies. In his way, we can use he colou o galaxies o classi y hem in o he RS, and hus wi h low sSFR, and BC, associa ed wi h high sSFR. Indeed, as onome s obse e his phenomenon in space, whe e small componen s o he RS, low-sSFR galaxies exis be ween wide componen s o he high-sSFR, BC galaxies. 31 32 4.1. Op ical galaxy classi ica ion Le us illus a e hese concep s wi h eal-li e obse a ions o galaxies collec ed om SkySe e ’s Sloan Digi al Sky Su ey, o SDSS. A da a se con aining 44,829 galaxies in he low edshi en i onmen was ob ained wi h p ecise spec oscopic (colou ) mea- su emen s along wi h an es ima ion o hei speci ic s a o ma ion a e, sSFR. Each indi idual galaxy was ep esen ed as a bi-dimensional da a poin in Figu e 4.1 using he colou indexes g− and −i as ea u es. The colou ing o each poin was calcula ed using he sSFR es ima ion in i s app op ia e as onomical o m (see [4]). Galaxies wi h a low speci ic s a o ma ion a e we e colou ed in ed, while hose wi h a high sSFR a e in blue. Figu e 4.1: Low edshi sample o 44,829 galaxies. Two clus e s a e clea ly isible. Figu e 4.1 demons a es ha he colou o galaxies is indeed co ela ed o hei s a o - ma ion ac i i y. The wo dis inc clus e s, co esponding o he blue cloud (BC) and he ed sequence (RS) galaxies, indica e ha classi ica ion o galaxies in o one o he abo e classes wi h a wo componen GMM using he colou s as ea u es is use ul o de e mine he sSFR le el. The e e ence pape o his sec ion [4] ou lines a new algo i hm, called Red D agon. This algo i hm slices he low edshi en i onmen s in o u he subg oups, applies a GMM o ob ain a pa ame iza ion o he componen s using he colou in o ma ion o each galaxy, and hen uses in e pola ion o e he disc e e mix u e model pa ame e s o ob ain a con inuous GMM model ac oss edshi . Classi ica ion o each indi idual galaxy is done, as discussed in Chap e 2, using he pos e io p obabili ies o each o he componen s, usually assigning i o he componen wi h he g ea es p obabili y o based on some h eshold. Howe e , Red D agon is a much mo e sophis ica ed algo i hm since i uses an addi ional wo colou indexes, o a ea u e space o ou dimensions. Each galaxy is he e o e ep esen ed by a colou ec o deno ed as ci. Chap e 4. Real li e applica ions 33 In addi ion, i conside s a “noise” co a iance ma ix o each galaxy, ∆i, o accoun o he in insic e o s in he measu emen s o he di e en colou bands. All in all, he likelihood o he K-componen Gaussian mix u e model is L(θ|S) = Ngal Y i=1 K X k=1 Lk(θk|ci),(4.1) whe e Ngal is he o al numbe o galaxies and Lk(θk|ci) is he likelihood o he i- h galaxy in he k- h componen , which is equal o Lk(θk|ci) = πk p(2π)Ngal |Σk+ ∆i|exp −1 2(ci−µk) (Σk+ ∆i)−1(ci−µk).(4.2) A e Red D agon has es ima ed he model’s pa ame e s con inuously ac oss edshi , galaxy classi ica ion is pe o med assigning each galaxy o he class ha maximizes he membe ship p obabili y, i.e, α= a g max α=1,...,K Lα(θα|ci) PK k=1 Lk(θk|ci).(4.3) Red d agon was es ed by he au ho s on he da a se ep esen ed in Figu e 4.1 o es ima e he model pa ame e s o a wo componen GMM in he al eady commen ed 4-dimensional colou space. Figu e 4.2 shows he esul s, as in he p e ious igu e, in he g− and −i axis, whe e he colou ep esen s he es ima ed p obabili y o each galaxy being om he RS ( ed sequence) class o galaxies, wi h hose wi h e y low p obabili y (and hence likely o be om he BC class) in blue. Figu e 4.2: F om “Red D agon, a edshi -e ol ing Gaussian mix u e model o galaxies” [4]. Because he clus e s in he diag am o e lap wi h hose in Figu e 4.1 co esponding o he low and high speci ic s a o ma ion a es, we can conclude ha classi ica ion based on he abo e GMM is a good p edic o , as sugges ed ea lie , o he le el o sSFR. 40 4.4. Conclusion 4.4 Conclusion Th oughou his disse a ion we ha e s udied he heo e ical amewo k o semi su- pe ised classi ica ion, wi h pa icula a en ion o he mix u e model and co- aining app oaches. A e pe o ming a numbe o simula ions o es he beha iou o hese echniques, we ha e seen a numbe eal li e use cases, anging om galaxy classi ica ion, o analysing ain occupancy da a. Semi-supe ised lea ning is s ill an unde -de eloped ield which would bene i om mo e igo ous heo e ical ounda ions. In addi ion, his lea ning pa adigm is subjec ed o he same challenges as all o he s in he con ex o machine lea ning, ha being he pace o ad ancemen s in he ield, which demands cons an inno a ion and adap a ion. Ne e heless, semi-supe ised classi ica ion has p o en o be an ex emely use ul lea n- ing pa adigm in he cu en con ex o la ge quan i ies o unlabelled da a being a ailable in o de o enhance classical supe ised classi ie s. The p ocess o de eloping his wo k has been ex emely ewa ding, as i has been a g ea excuse o di e deep in o opics ha we e ou o he scope o his deg ee, and ead sou ces ha we e o g ea in e es bo h o his disse a ion and in a b oade sense o my de elopmen as a scien is and as a human. Appendix A R and Py hon code R and Py hon we e used ex ensi ely, especially o chap e s 3 and 4. No included in his disse a ion o main ain b e i y, he code is p o ided sepa a ely in he CODE olde . We p o ide a basic guide o na iga e h ough hese iles. •3.1 In oduc ion -gMM.R •3.2 Fi s se o simula ions -Simula ion 1.R •3.3 Second se o simula ions -Simula ion 2.R •3.4 Example: e ec s o an inco ec model -Simula ion 3.R •3.5 A co- aining simula ion -Simula ion CoT aining.R •4.1 Op ical galaxy classi ica ion -Galaxies ( olde ) –galaxies.R –SDSS low z sample.cs (da a base) •4.2 Case s udy: ain occupancy da a -T ain occupancy analysis ( olde ) – ain da a.R –Loading Da a.cs (da a base) •4.3 Sel - aining: a inal semi-supe ised app oach -Sel -T aining ( olde ) –main.py –plo .py –da ase .cs (da a base) 41 Bibliog aphy [1] Abney, S. (2002) Boo s apping. P oceedings o he 40 h Annual Mee ing o he Associa- ion o Compu a ional Linguis ics (ACL), July 2002, pp. 360-367. [2] Benaglia, T., Chau eau, D., Hun e , D.R., Young, D.S. (2009) “mix ools: An Package R o Analyzing Fini e Mix u e Models.” Jou nal o S a is ical So wa e, 32(6), 1–29. h p://www.js a so .o g/ 32/i06/. [3] B ´emaud, P. (2017) Disc e e P obabili y Models and Me hods, Sp inge . [4] Black, W.K., E a d, A. (2022) Red D agon: a edshi -e ol ing Gaussian mix u e model o galaxies, Mon hly No ices o he Royal As onomical Socie y, Volume 516, Issue 1, Oc obe 2022, Pages 1170–1182, h ps://doi.o g/10.1093/mn as/s ac2052. [5] Blum, A., Mi chell, T. (1998) Combining labeled and unlabeled da a wi h co- aining. In: P oceedings o he 11 h Annual Con e ence on Compu a ional Lea ning Theo y, As- socia ion o Compu ing Machine y. [6] Bouguila, N., Fan, W. (Eds.) (2020) Mix u e Models and Applica ions, Sp inge Na u al Swi ze land. [7] Helmbold, D., Williamson, B. (Eds.) (2001) 14 h and 5 h Annual Con e ence on Compu- a ional Lea ning Theo y, Ams e dam, The Ne he lands, July 16-19, 2001 p oceedings. [8] Jo, T. (2021) Machine Lea ning Founda ions: Supe ised, Unsupe ised, and Ad anced Lea ning, Sp inge . [9] P esno, M. A. (2023) Polic´ıa p edic i a y p e enci´on de la iolencia de g´ene o: el sis ema VioG´en. Re is a de los Es udios de De echo y Ciencia Pol´ı ica, UOC, no iemb e, 2023. [10] Sasaki, T. (2022) Semi-supe ised classi ica ion on a ex da ase , Kaggle, www.kaggle.com/code/sasaki e suya/semi-supe ised-classi ica ion-on-a- ex -da ase [11] Schwenke , F., T en in, E. (Eds.) (2011) Pa ially Supe ised Lea ning, Fi s IAPR TC3 Wo kshop, PSL 2011 Ulm, Ge many, Sep embe 15-16, Re ised Selec ed Pape s, Sp inge . [12] Zhou, Z.-H. (2013) Unlabeled Da a and Mul iple Views. In: Pa ially Supe ised Lea n- ing: Second IAPR In e na ional Wo kshop, PSL 2013, Nanjing, China, May 13-14, 2013. [13] Zhu, X., Goldbe g, A. B. (2009) In oduc ion o Semi-Supe ised Lea ning, Mo gan & Claypool. 43