Deep transfer learning-based gaze tracking for behavioral activity recognition

Deep ans e lea ning-based gaze acking o beha io al ac i i y ecogni ion Ja ie de Lope a, ⇑ , Manuel G aña b a Depa men o A i icial In elligence, Uni e sidad Poli écnica de Mad id (UPM), Mad id, Spain b Compu a ional In elligence G oup, Uni e si y o he Basque Coun y (UPV/EHU), San Sebas ian, Spain a icle in o A icle his o y: Recei ed 7 Ma ch 2021 Re ised 8 June 2021 Accep ed 25 June 2021 A ailable online 23 May 2022 Keywo ds: Deep ans e lea ning Gaze acking Gaze e hog am Human ac i i y ecogni ion abs ac Compu a ional E hology s udies ocused on human beings is usually e e ed as Human Ac i i y Recogni ion (HAR). Speci ically, his pape belongs o a line o wo k on he iden i ica ion o b oad cogni- i e ac i i ies ha use s ca y ou wi h compu e s. The keys one o his kind o sys ems is he nonin asi e de ec ion o he subjec ’s gaze ixa ions in selec ed display a eas. Nonin asi eness is ensu ed by using he con en ional lap op came as wi hou addi ional illumina ion o acking de ices. The gaze e hog ams, composed as sequences o gaze ixa ions, a e he basis o iden i y he use ac i i ies. To de e mine he gaze ixa ion display a eas wi h he highes accu acy, his pape explo es he use o a ans e lea ning app oach applied o se e al well-known deep lea ning ne wo k (DLN) a chi ec u es whose inpu is he eye a ea ex ac ed om he ace image,and ou pu is he iden i ica ion o he gaze ixa ion a ea in he compu e sc een. Two di e en da ase s a e c ea ed and used in he alida ion expe imen s. We epo encou aging esul s ha may allow he gene al use o he sys em. Ó2022 The Au ho (s). Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/). 1. In oduc ion Compu a ional E hology [1] has become a ho esea ch ield in he las ew yea s. I in eg a es he in o ma ion om se e al di e - en senso s and ac i i y measu emen de ices in o de o cha ac- e ize he beha io o li ing beings. Speci ically, he compu e - based analysis and ecogni ion o human beha io , e e ed o as Human Ac i i y Recogni ion (HAR) [2], ecei es plen y o a en ion and con ibu ions. Basically, he e a e wo ypes o senso s used in HAR esea ch: came as [3] and ine ial senso s [4]. In compu a- ional neu oe hology, hese senso s a e usually combined wi h neu onal ac i i y da a cap u ed by using, o example, elec oen- cephalog aphy (EEG) equipmen [5]. Much e o in HAR esea ch is cu en ly di ec ed o he moni o ing o aging people [14], and o he pe o mance imp o emen in some spo s [15]. The moni- o ing o elde ly people is usually mo i a ed by beha io al decline due o neu odegene a i e diseases and i s goals is o de ec abno - mal si ua ions o aise ala ms [6], o example, all de ec ion [16]. HAR s udies a e usually o ien ed o he iden i ica ion o low le el ac i i ies, o ins ance, he de ec ion o abno mal beha io al si ua- ions in he elde ly [6] by he use o 3D skele on models o body pos u es [7], hence hey do no deal wi h highe le el beha io ep- esen a ions such as p o ided by e hog ams. An e hog am is a ime plo o he low le el ac ions ca ied ou by he subjec unde obse a ion ha p o ides a high le el beha - io al ep esen a ion. E hog ams ha e been used o animal pheno- ype cha ac e iza ion [8]. We a e cu en ly in e es ed in he cha ac e iza ion o beha io al s a es o a lap op compu e use by using he lap op came a and he mic ophone o de e mine he ac i i y pe o med by he use by nonin asi e compu a ional me hods. P e iously, we ha e s udied he pe o mance o con en- ional machine lea ning app oaches on such ask [9]. In his pape we explo e he use o deep lea ning echniques o ecognize he subjec ’s beha io al ac i i y. Ou hypo hesis is ha he subjec ’s gaze ixa ions in o ma ion allows o de e mine he speci ic ac i i- ies in which he subjec is engaged [9,61]. A isual ixa ion is he sus ained gaze du ing a ime in e al in a speci ic di ec ion which alls upon a single loca ion in he isual s imulus. I s a e age du a ion in uncon olled condi ions is abou 200 ms [12]. The saccades a e quick, simul aneous mo emen s o bo h eyes be ween wo o mo e phases o ixa ion in he same di ec ion [13].Blinking is he semi-au oma ic apid closing o he eyelids. I s a e is gene ally g ea e han a dozen blinks pe minu e, al hough i may dec ease when he eyes a e ocused on an objec o an ex ended pe iod o ime, o example, when eading. h ps://doi.o g/10.1016/j.neucom.2021.06.100 0925-2312/Ó2022 The Au ho (s). Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/). ⇑ Co esponding au ho . E-mail add esses: [email p o ec ed] (J. de Lope), [email p o ec ed] (M. G aña). Neu ocompu ing 500 (2022) 518–527 Con en s lis s a ailable a ScienceDi ec Neu ocompu ing jou nal homepage: www.else ie .com/loca e/neucom As he in o ma ion is e ie ed du ing he ixa ions, we de e - mine when hey a e p oduced and in which o de hey a e pe - o med. We call gaze e hog ams hese empo al sequences o isual ixa ions which a e he a omic ac ions building up he beha io al ep esen a ion. We de ine a eas o he display which ecei es he use a en ion in o de o ca ego ize he isual ixa- ions. The gaze e hog am may be used o ecognize he subjec ’s beha io al ac i i y. The wo k in his pape is de o ed o he e alu- a ion o deep lea ning a chi ec u es on he ask o ecognizing he gaze ixa ion om un il e ed images o he eye egion. The es o he pape is o ganized as ollows. Fi s , Sec ion 2p o- ides a sho iew o he s a e-o -a in bo h lines o ou wo k: gaze de ec ion and acking, and deep lea ning echniques applied o li e sciences. Sec ion 3desc ibes he expe imen al da ase s and he p oposed compu a ional me hods expe imen ed wi h. Sec ion 4 p o ides he expe imen al esul s. The e, we also o e a c i ical discussion on he esul s. Finally, Sec ion 5we conclude wi h some summa izing ema ks on ou wo k and di ec ions o u u e wo k. 2. Backg ound This sec ion p o ides a sho e iew o he s a e-o -a in ela ed esea ch. Fi s , we summa ize he wo ks on gaze de ec ion and acking wi h app oxima e o equi alen goals. Then, we e iew some an eceden s and cu en de elopmen s in he g ow- ing a ea o deep lea ning applied o li e sciences. 2.1. Gaze de ec ion and acking Gaze in o ma ion has been used o diagnos ic and ac i e in e - ac ion pu poses [10,11,18]. Fo example, gaze in e ac ion has been used o communica ion wi h people su e ing ex eme disabili y [24] o in pa ien s wi h Alzheime ’s Disease (AD) [25]. Diagnos ic applica ions ha e been widesp ead in many di e en a eas such as neu oscience [26,27], in luence o s uden s’ isual a en ion and school ailu e [28] o analysis o acial exp ession explo a ion in subjec s wi h social anxie y [29]. Gaze de ec ion has been a esea ch challenge o a long ime [17,18]. Ea ly success ul app oaches [19] we e based on elec- ooculog aphy (EOG), which is a echnique ha uses a se ies o elec odes si ua ed in he use ’s ace o measu e he eye mo ion in an elec omagne ic ield. Videooculog aphy (VOG) sys ems [20] a e op ical-based sys ems using speci ic illumina ion sys ems —o en in a ed— ha enhance he de ec ion o eye ea u es such as he pupil and he co nea. The e is a need o much less in asi e sys ems ha do no equi e he subjec o wea speci ic in usi e echnology, as is he case o EOG and VOG. Solu ions based on compu e ision use con en ional machine lea ning echniques, some a e based on he localiza ion o he eyeb ows [21], o he s use he es ima ion o he 3D ace mo ion om a single came a [22]. Recen app oaches based on deep lea ning a chi ec u es ha e been es ed in neu o- science s udies [23]. The wo k in his pape goes in his di ec ion owa ds minimally in asi e eliable gaze de ec ion and acking sys ems. 2.2. Deep lea ning in he li e sciences Deep Lea ning (DL) app oaches a e he p o agonis o A i icial Neu al Ne wo k (ANN) esu gence in he las decade [31–33]. They o e come he p oblem o he anishing g adien and o e i ing by a ious app oaches. They p oduce a da a d i en hie a chy o abs ac ep esen a ions by s ochas ic g adien descen aining p ocedu es. Speci ically, he con olu ional neu al ne wo k (CNN) [35] aining p oduces a hie a chy o il e s uned om he da a. CNNs ha e been ex emely in luen ial in he ad ance o compu e ision (CV) asks. This a chi ec u e has inspi ed new gene a ions o DL ne wo ks (DLN) wi h di e se a chi ec u es, which a e epo ing supe io pe o mance on many di e en p oblems in a eas such as image p ocessing [36,37], pa e n ecogni ion and objec de ec ion [38–40], classi ica ion [41,42], acking [48], and ac i i y ecogni- ion om da a p o ided by ine ial senso s [49]. In he Li e Sciences (LS) he numbe o epo ed DLN applica- ions du ing he las i e yea s ha e been g owing exponen ially [34]. Example applica ions o DLN in LS a eas a e he analysis o medical images in he neu osciences [23,43] and o he medical a eas [44,45] including ea ly s age de ec ion o COVID-19 in X- ay imaging [46,47]. DLNs ha e been also applied o acial image p ocessing, which is a a he complex objec because o many di - e en ac o s like he ace posi ion and o ien a ion, he mou h and eyes opening, and he human skin colo ange. The e a e epo ed DLN app oaches o ace con ou de ec ion [50,51], he acial com- ponen s ex ac ion [52,53], biome ic acial ecogni ion [54], and gende classi ica ion [55]. 3. Ma e ials and me hods As p e iously s a ed, ou se up employs he beha io al ac i i y ecogni ion sys em o de e mine he ac i i y ca ied ou by he subjec [9]. This sys em uses gaze e hog ams o desc ibe and iden- i y such ac i i ies. Fig. 1(a) shows an ins ance o a gaze e hog am ob ained om a subjec eading a ex on he compu e display. Fo ac i i y ecogni ion pu poses i is enough ha he gaze acking sys em iden i ies he gaze ixa ion a ge s co esponding o he b oad a eas in Fig. 1(b). The a ge numbe o de has been a bi a - Fig. 1. (a) Gaze e hog am co esponding o he use ac i i y ‘‘ eading a ex ” in an expe imen o du a ion 200 s. The a ge s co espond o nine di e en display a eas in which he subjec ’s ixa ions a e de ec ed. (b) Templa e used o calib a ion. The numbe s deno e he sequence o loca ions o he a ge a eas o gaze ixa ions ollowed by he use du ing calib a ion. The same numbe s a e used as ou pu ca ego ies o he DLNs. J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 519 ily de ined in o de o educe he subjec a igue while pe o ming he sys em calib a ion. We desc ibe he o e all cogni i e ac i i y ecogni ion based on gaze e hog ams elsewhe e [9]. The wo k in his pape co e s a no el p oposal ha u ilizes deep lea ning o es ima ing he gaze ixa ion on he isual a ge a eas. The sys em ha dwa e con igu a- ion is a lap op compu e endowed wi h a web came a on op o he sc een upon which a use is wo king. The dis ance o he ace o he came a is oughly 50 cm, and he came a iew o he ace is on al, al hough he subjec can mo e eely and change pose a will. We a e using o he shel web came as ha a e ac o y ins alled in lap ops, he e o e obus ness is a challenge and a lim- i a ion. The esolu ion o hese came as is limi ed and o en he image quali y is qui e low. Addi ional di icul ies a ise om he uncon olled illumina ion condi ions, and he use eedom o mo emen in on o he came a. 3.1. Da ase s We ha e gene a ed wo di e en da ase s o hese expe i- men s. Bo h a e p oduced om lap op came a cap u ed ideos wi h esolu ions o 720p, in which he subjec s pe o m ixa ions in o de on e e y a ge o a calib a ion empla e o 3 s. Then, hose images a e selec ed o emo e examples wi h oo blu ed o e y simila images and unclea a ge des ina ions and hey a e hand-labeled o assign he a ge o each one. The i s da ase con ains images om 12 subjec s wi h di e - en equipmen and illumina ion condi ions. The images in his da ase ha e been balanced in o de o gua an ee an equi alen numbe o images in each class, ying o an icipa e oubles du ing he aining s age. This da ase con ains 450 images. The second da ase con ains images om a unique subjec . The ideos ha e been eco ded unde di e en illumina ion condi- ions and a ying dis ance o he came a. The unde lying idea is o compa e he pe o mance be ween ANNs ained wi h gen- e al, mul i-use da a and ailo ed, single-use da a. This da ase is composed by 700 images. To localize he ace in he images we apply a p e- ained de ec- o based on his og ams o o ien ed g adien s (HOG) [56] as inpu ea u es o classi ica ion by linea suppo ec o machines (SVM) [57]. Once he ace is localized in he image, he nex s ep consis s o de e mining he posi ion o ace landma ks. This p oblem is known as ace alignmen . We use a p e iously ained ensemble o eg ession ees o es ima e he ace landma k posi ion di ec ly om a spa se subse o pixels in ensi ies [60]. The me hod e u ns 68 2D poin s in he image ha can be used o localize he eyes, eye- b ows, nose, mou h, and jawline. This app oach allows almos eal- ime esponse, al hough we ha e ound ouble when he use is wea ing some kind o glasses du ing he da a cap u e. Finally, we selec he eye a ea in he o iginal images o add hem o he ali- da ion da ase s. Fig. 2 shows some examples o hose images and he co esponding label. In his case we use he a ge iden i ie me hod o labeling [61]. 3.2. Deep ans e lea ning We ha e e ained six models o DLN. We use a ans e lea ning app oach, whe e each DLN has been p e iously ained o e da a om he ImageNe challenge. We keep he weigh s o he in e media e laye s, e aining he inal laye s ha p oduce he ac ual classi ica ion ou pu . In deep ans e lea ning [30] he al eady ained DLN hidden laye s a e assumed as a gene al ea u e ex ac ion p ocedu e, de ining a mani old ha can be used o map he inpu da a o classi ica ion o eg ession asks ha a e di e en om he o iginal one. Task speci ic in o ma ion is p o ided when aining he ou pu laye s o he DLN. We use he same ou pu ully connec ed laye o all he ne s. I con ains 10 neu ons, each one iden i ies one o he display a ge a eas. The en h neu on is used o de ec cases in which he subjec has he eyes closed. We ha e chose a so max unc ion o compu - ing he classes due o i is usually ecommended o likelihood compu a ion in mul i-class domains. Unless s a ed o he wise, we ha e used he Adam op imize wi h a lea ning a e o 10 4 .We ha e alida ed he e ained models by c oss- alida ion in all he expe imen s epo ed in he nex sec ion. We ha e epea ed 30 imes a 80% hold ou alida ion, whe e we by andomly selec a 80% o he da ase s o aining and use he emaining 20% o es . We epo he a e age accu acy o he es esul s. 3.2.1. VGG19 The i s DLN e alua ed was he classical VGG19 [62].I isa classical CNN which has 19 con olu ional laye s ollowed by max-pooling laye s o educe he image size. In o de o adap he model o ou da a we ha e emo ed he las laye s o he p e- ained ne wo k, and added and ained wo ully connec ed laye s wi h 50 and 25 neu ons, espec i ely, wi h i s owns d opou laye s o educe he o e i ing. 3.2.2. Incep ion- 3 The Incep ion neu al ne wo k [63] has se e al e sions, he ou h is he mos ecen . We ha e used he p e ious e sion because o i s a ailabili y. I s s uc u e is composed o a pa e n o laye s ha is eplica ed along he ne . The e a e modules wi h mul iple con olu ional laye s in pa allel ha ex ac di e en image ea u es, which a e conca ena ed a he end o he module. We ha e added and ained an addi ional ully connec ed laye o i i o ou da ase s. 3.2.3. Xcep ion The Xcep ion neu al ne wo k [64] uses he same modula com- posi ion idea o Incep ion a chi ec u es bu he e is a modi ica ion in he pa e ns: i changes he pa allel con olu ional laye s by sep- a able con olu ional laye s. These new laye s allow o educe he compu a ions, being he ime equi ed o ain much mo e images, conside ably sho e . The Xcep ion s uc u e p esen s h ee di e en s ages. The ini- ial s age applies a il e o he image o educing he image size while i keeps he con olu ional laye s. The middle s age a e epea ed modules, which a e duplica ed up o eigh imes. The inal s age has been modi ied and e ained o adap i o ou da ase s. He e we ha e used wo ully connec ed laye s wi h 50 and 20 neu- ons ollowed by a single d opou laye o a oid o e i ing. 3.2.4. ResNe 50 ResNe 50 [65] is a esidual ne wo k. This kind o DLN a chi ec- u e ies o model he esidual o he p edic ion a p e ious laye s. I has di ec p opaga ion o he inpu along he laye s o he ne - wo k in o de o compu e his esidual. This s uc u al ea u e alle- ia es he anishing g adien p oblem and p o ides in e es ing compu a ional p ope ies, such ha he compu a ion a a gi en laye is independen om p e ious laye s. Residual ne wo ks may ha e a e y la ge numbe o laye s, he one ha we e ain on ou da ase s has 50 laye s ha a e g ouped in o se e al blocks. A he beginning o each block, he compu ed esidual is s o ed and i is used a he end o he block wi h he compu ed weigh s. In his case, we a e using he SGD op imize due o i s supe io pe o - mance agains o he op imize s o his kind o ne wo ks. The lea ning a e is 10 5 and he decay a e is 10 6 o each i e a ion. J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 520 Fig. 2. Examples o eye egion images cap u ed while subjec s a e pe o ming ixa ions in each a ge ha compose he da ase s. These images a e he inpu o he DLNs p o iding he gaze ixa ion iden i ica ion. Fig. 3. Accu acy cu es in aining and es wi h he mul i-use da ase . J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 521 3.2.5. Incep ion-ResNe - 2 The Incep ion-Resne ne [66,67] combine bo h ResNe and Incep ion app oaches o c ea e a model wi h he ad an ages p o ided by hem. The s uc u e is composed by se e al blocks besides he pa allel con olu ional laye s used o conca ena e blocks. Inside he blocks he e a e epea ed modules o he Incep- ion la o . The e is also connec ions om he beginning o blocks o he end simila o ResNe ones. 3.2.6. DenseNe DenseNe [68] ollows a design idea simila o ResNe al hough now au ho s add he esidual o each block globally and no only in he end o each block. Thus, i appea s se e al connec ions om he inpu s he con olu ional laye s in each block o he ou pu s o o he blocks. Thanks o his modi ica ion he ne is mo e compac and equi es lesse laye s o ex ac in o ma ion om he image because o each laye can ecei e in o ma ion om p e ious laye s. Fig. 4. Ca ego ical c oss-en opy loss cu es in aining and es wi h he mul i-use da ase . Table 1 Ne s accu acy and e o wi h he mul i-use da ase . Ne Bes Accu acy Lowes E o VGG19 89.01% 0.3452 Incep ion- 3 86.96% 0.5529 Xcep ion 82.61% 0.6023 ResNe 50 86.96% 0.6531 Incep ion-ResNe - 2 84.78% 0.6248 DenseNe 91.30% 0.4281 J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 522 Table 2 No malized con usion ma ix om DenseNe and he mul i-use da ase . P edic ed Ta ge 0123456789 Ac ual Ta ge 0 100 000 0 000 1010 000 0 000 20 0 .833 0 0 0 0 0 0 .167 30 0 .143 .857 0 0 0 0 0 0 4000 010 0 000 50 0 0 0 0 .667 0 0 0 .333 60 0 0 0 0 0 .800 .200 0 0 7000 000 0 100 800 0 0 0 0 0 0 1 0 9000 000 0 001 Fig. 5. A e age accu acy cu es in aining and es o e he single-use da ase . J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 523 4. Expe imen al esul s and discussion Now we discuss he lea ning esul s and gene al pe o mance ob ained wi h he DLNs desc ibed abo e. Fi s , we show he mul i-use da ase esul s. Then, we summa ize and compa e he esul s ob ained on he single-use da ase . 4.1. Mul i-use da ase Fig. 3 depic s he a e age accu acy cu es ob ained in aining and es phases wi h he mul i-use da ase . All he ne s achie e a high accu acy a e a low numbe o epochs. Mo eo e , he en- dency in bo h aining and es a e almos pa allel in e e y case. The ea u es ex ac ed by he p e ained hidden laye s o he DLNs appea o p o ide a good baseline o his p oblem and ou da ase . Wi h he excep ion o VGG19, he DLNs s all a e oughly 30 Fig. 6. A e age ca ego ical c oss-en opy loss cu es du ing aining and es o e he single-use da ase . Table 3 Bes accu acy and e o achie ed by DLN models o e he single-use da ase . Ne Bes Accu acy Lowes E o VGG19 94.62% 0.1788 Incep ion- 3 93.55% 0.3151 Xcep ion 93.55% 0.2475 ResNe 50 91.40% 0.2486 Incep ion-ResNe - 2 91.24% 0.2982 DenseNe 95.70% 0.2195 J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 524 epochs. I usually comes om he ac ha DLNs s a o o e i o he aining da a. The e o e, p obably hey would need mo e da a o keep imp o ing he aining. T aining minimizes he ca ego ical c oss-en opy loss in o de o compa e he eal dis ibu ion wi h he p edic ed one. The lowe he ou pu o his unc ion, he g ea e he deg ee o simila i y o bo h dis ibu ions, and he g ea e he expec ed accu acy o classi- ica ion. Fig. 4 show he a e age e olu ion o his e o measu e in bo h aining and es da ase s. These cu es a e highly nega i ely co ela ed o he ones in Fig. 3: he highe he accu acy, he lowe he e o . He e, we also obse e ha he bes aining e olu ion esul s a e ob ained wi h he VGG19 ne . O he ne s end o o e i a e he i s epochs. Table 1 shows he highes accu acy and he lowes e o achie ed by e e y DLN a chi ec u e. The esul s indica e ha we ha e been able o ge a leas an accu acy o 80% wi h he new da a. DenseNe achie es he bes esul wi h mo e han 90%. Table 2 shows a ypical es con usion ma ix ob ained by he e ained DenseNe on he mul i-use da ase . Usually he DLNs end o ou pu e oneous a ge s when subjec s look a he a ge a eas loca ed a he bo om o he display. The eason is ha he eyes a e o en closed in hose images so ha i is ha d o de e mine he igh a ge unde hese condi ions. 4.2. Single-use da ase The p oposed ans e lea ning a chi ec u es achie e be e esul s when hey a e e alua ed on he second da ase , which is composed o images om a single use . The lea ning p oblem appea s easie han in he mul i-use case, because we emo e he da a a iabili y due o he use . Mo eo e , he da ase is la - ge han in he mul i-use case. We can obse e in Fig. 5 ha all DLN models achie e an accu acy g ea e han 80% in jus a ew epochs. Howe e , o e i ing appea s in he ini ial epochs so ha e aining he DLNs do no imp o e hei pe o mance anymo e. The hypo hesized eason is he high simila i y be ween all he images in he da ase . I migh be pa ially sol ed by emo ing edundan da a o by applying addi ional egula iza ion me hods apa om he d opou laye s deployed a design ime. Fig. 6 shows he e olu ion o he loss unc ion on lea ning and es da ase s. The e o a e alls apidly bu i emains s able a e he i s epochs. Table 3 summa izes he highes accu acy and he lowes e o achie ed by e e y DLN a chi ec u e a e ans e lea ning. The esul s a e be e han he ob ained wi h he mul i-use da ase . All he DLN a chi ec u es achie e accu acies o e 90%. DenseNe p o ides he bes esul s. No e ha he e he ne s a e lea ning o classi y he gaze co esponding o jus one subjec . This gi es an idea abou how impo an can be o ailo he classi ie s o a inal use . Table 4 shows a ypical con usion ma ix gene a ed om Den- seNe and he single-use da ase . The con usion e o in p edic ed a ge s ollows a e y simila pa e n o he mul i-use case. 4.3. Discussion The models achie e compe i i e esul s wi h bo h da ase s. The es accu acy achie ed o e he single-use da ase is g ea e bu hese esul s mus be aken wi h ca e. The mul i-use case could o e he be e solu ion o a global sys em o de aul mode, while he single-use case has o be e ained o each pa icula use . We expec ha he esul s wi h he mul i-use da ase should be imp o ed i mo e images om new use s a e added o he da a- se because he cu en numbe o images is no pa icula ly high and DLN me hods usually equi e la ge da ase s o e ec i e aining. Also we ha e used he same s uc u e and laye s in ne s o bo h da ase s. P obably we could modi y some laye s in o de o manage he o e i ing p oblems ound wi h he single-use da a- se , as p e iously commen ed. 5. Conclusions and u he wo k We ha e p esen ed a me hod o gaze ixa ion de ec ion based on deep ans e lea ning in he con ex o beha io al ac i i y ecogni ion sys ems. This is usually an impo an pa o such sys- ems. In ou case we mus achie e he bes pe o mance o he gaze acking sys ems because he goal o ou sys em is o de e mine ac i i ies ha a use ca ies ou in on a compu e and he inpu s come om he came a on op o he sc een. In spi e o he educed da ase s used in he expe imen s he use o a ailable public p e- ained ne wo ks o domain ans e lea n- ing allows o achie e good pe o mance wi h a o dable compu a- ional cos . The bes esul s acco ding he ecogni ion accu acy ha e been epo ed by he DenseNe model. O he models equi e lowe aining ime o a e easie o implemen , so i should be con- side ed as jus one i em o conside . Fu u e wo ks will check inno a i e ecen DL. Speci ically, he ecommenda ion o he e iewe s conce ning he 3D-ResNe 35 a chi ec u e [69,70] ha p omises enhanced esul s due o i s abil- i y o p ocess 3D da a. Ano he al e na i e o u u e wo k is o c e- a e a new a chi ec u e om sc a ch. We should ex end ou da ase o his endea o , because a basic equi emen o DLN aining a e la ge da ase s. CRediT au ho ship con ibu ion s a emen Ja ie de Lope: Concep ualiza ion, Me hodology, So wa e, W i ing – e iew & edi ing. Manuel G aña: Me hodology, W i ing – e iew & edi ing. Table 4 No malized con usion ma ix om DenseNe o e he single-use da ase . P edic ed Ta ge 0123456789 Ac ual Ta ge 0 1000000000 10100000000 20010000000 30001000000 40 0 0 0 .937 0 0 0 .063 0 50 .040 0 0 0 .800 .120 0 0 .040 60000001000 7.133 0 0 0 0 0 0 .867 0 0 80000000010 90000000001 J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 525 Decla a ion o Compe ing In e es The au ho s decla e ha hey ha e no known compe ing inan- cial in e es s o pe sonal ela ionships ha could ha e appea ed o in luence he wo k epo ed in his pape . Acknowledgmen s This wo k has been suppo ed by FEDER unds h ough MINECO p ojec TIN2017-85827-P. This p ojec has ecei ed unding om he Eu opean Union’s Ho izon 2020 esea ch and inno a ion p o- g amme unde he Ma ie Sklodowska-Cu ie g an ag eemen No. 777720. XinZhe Jin con ibu ed some ea ly compu a ional expe iences. Re e ences [1] D.J. Ande son, P. Pe ona, Towa d a science o Compu a ional E hology, Neu on 84 (2014) 18–31. [2] M. V igkas, C. Nikou, I. Kakadia is, A e iew o human ac i i y ecogni ion me hods, F on . Robo . A i . In ell. 2 (2015) 11. [3] S.-R. Ke, H. Le Uyen Thuc, Y.-J. Lee, J.-N. Hwang, J.-H. Yoo, K.-H. Choi, A e iew on ideo-based human ac i i y ecogni ion, Compu e s 2(2) (2013) 88–131.. [4] J.Y. Yang, J.S. Wang, Y.P. Chen, Using accele a ion measu emen s o ac i i y ecogni ion: An e ec i e lea ning algo i hm o cons uc ing neu al classi ie s, Pa e n Recogn. Le . 29 (16) (2008) 2213–2220. [5] M. G aña, M. Aguila -Mo eno, J. De Lope, I. Baglie o, X. Ga mendia, Imp o ed ac i i y ecogni ion combining ine ial mo ion senso s and elec oencephalog am signals, In . J. Neu al Sys . 30 (10) (2020) 2050053. [6] A. Len zas, D. V akas, Non-in usi e human ac i i y ecogni ion and abno mal beha io de ec ion on elde ly people: A e iew, A i . In ell. Re . 53 (2020) 1975–2021. [7] N. Tasnim, M. Islam, J.-H. Baek, Deep lea ning-based ac ion ecogni ion using 3D skele on join s in o ma ion, In en ions 5 (2020) 49. [8] J.H.F. Abeelen, Mouse mu an s s udied by means o e hological me hods, Gene ica 34 (1964) 79–94. [9] J. De Lope, M. G aña, Beha io al ac i i y ecogni ion based on gaze e hog ams, In . J. Neu al Sys . 30 (7) (2020) 2050025. [10] A. Geo ge, Image based eye gaze acking and i s applica ions. a Xi 2019, 1907.04325.. [11] R. Ho , How do you Google? New eye acking s udy e eals huge changes, Fo bes Online, 2015. [12] B. Cassin, S. Solomon, Dic iona y o Eye Te minology, T iad Publising Company, Gaines ille, Flo ida, 1990. [13] J.D. Ende le, D.A. Sie a, A new linea muscle ibe model o neu al con ol o saccades, In . J. Neu al Sys . 73 (2013) 1350002.. [14] R.G. Hussain, M.A. Ghazan a , M.A. Azam, U. Naeem, S.U. Rehman, A pe o mance compa ison o machine lea ning classi ica ion app oaches o obus ac i i y o daily li ing ecogni ion, A i . In ell. Re . 52(1) (2019) 357– 379.. [15] G. And ienko, N. And ienko, G. Budziak, J. Dykes, G. Fuchs, T. on Landesbe ge , H. Webe , Visual analysis o p essu e in oo ball, Da a Min. Knowl. Disc. 31 (6) (2017) 1793–1839. [16] E.E. S one, M. Skubic, Unob usi e, con inuous, in-home gai measu emen using he Mic oso Kinec , IEEE T ans. Biomed. Eng. 60 (10) (2013) 2925– 2932. [17] A.T. Duchowski, Eye T acking Me hodology — Theo y and P ac ice, Sp inge , Cham, 2017. [18] A.T. Duchowski, Gaze-based in e ac ion: A 30 yea e ospec i e, Compu . G aph. 73 (2018) 59–69. [19] L.R. Young, D. Sheena, Su ey o eye mo emen eco ding me hods, Beha . Res. Me hods Ins um. 7 (5) (1975) 397–439. [20] B.W. Blakley, L. Chan, Me hods conside a ions o nys agmog aphy, J. O ola yngol. Head Neck Su g. 44 (2015) 25. [21] L. Flo ea, C. Flo ea, C. Ve an, Recogni ion o he gaze di ec ion: Ancho ing wi h eyeb ows, J. Vis. Commun. Image Rep. 35 (2016) 67–77. [22] K.R. Pa k, J.J. Lee, J. Kim, Gaze posi ion de ec ion by compu ing he h ee dimensional acial posi ions and mo ions, Pa e n Recogn. 35 (11) (2002) 2559–2569. [23] Y.-H. Yiu, M. Aboula a, T. Raise , L. Ophey, V.L. Flanagin, P. Zu Eulenbu g, S.-A. Ahmadi, Deep- og: Open-sou ce pupil segmen a ion and gaze es ima ion in neu oscience using deep lea ning, J. Neu osci. Me hods 324 (2019) 108307. [24] N. Ba ba a, T.A. Camille i, K.P. Camille i, EOG-based eye mo emen de ec ion and gaze es ima ion o an asynch onous i ual keyboa d, Biomed. Signal P ocess. Con ol 47 (2019) 159–167. [25] P.M. Insch, G. Slesso , J. Wa ing on, L.H. Phillips, Gaze de ec ion and gaze cuing in Alzheime ’s Disease, B ain Cogn. 116 (2017) 47–53. [26] O. G ynszpan, J. Bou eille , S. G ynszpan, F. Le Ba illie , J.C. Ma in, J. Nadel, Al e ed sense o gaze leading in au ism, Res. Au ism Spec . Diso d. 67 (2019) 101441. [27] J. Kim, J. Seo, T.H. Laine, De ec ing bo edom om eye gaze and EEG, Biomed. Signal P ocess. Con ol 46 (2018) 302–313. [28] M.-J. Tsai, H.-T. Hou, M.-L. Lai, W.-Y. Liu, F.-Y. Yang, Visual a en ion o sol ing mul iple-choice science p oblem: An eye- acking analysis, Compu . Educ. 58 (2012) 375–385. [29] A. Gu ie ez-Ga cia, A. Fe nandez-Ma in, M. Del Libano, M.G. Cal o, Selec i e gaze di ec ion and in e p e a ion o acial exp essions in social anxie y, Pe s. Indi id. Di e . 147 (2019) 297–305. [30] M. Talo, U.B. Baloglu, O. Yildi im, U.R. Acha ya, Applica ion o deep ans e lea ning o au oma ed b ain abno mali y classi ica ion using MR images, Cogn. Sys . Res. 54 (2019) 176–188. [31] Y. Bengio, P. Lamblin, D. Popo ici, H. La ochelle, U. Mon eal, G eedy laye - wise aining o deep ne wo ks (2007) 19. [32] G.E. Hin on, D. Osinde o, Y-W. Teh, A as lea ning algo i hm o deep belie ne s, Neu al Compu . 18(7) (2006) 1527–1554.. [33] M.A. Ranza o, Y.-L. Bou eau, Y. LeCun, Spa se ea u e lea ning o deep belie ne wo ks, Con . Neu al In . P oc. Sys . (2007) 1185–1192. [34] D. Bacciu, P. Lisboa, J. Ma in-Gue e o, R. S oean, A. Vellido, Bioin o ma ics and medicine in he e a o deep lea ning, 2018, a Xi :1802.09791.. [35] Y. Lecun, P. Ha ne , L. Bo ou, Y. Bengio, Objec ecogni ion wi h g adien - based lea ning, in: Shape, Con ou and G ouping in Compu e Vision. Lec u e No es in Compu e Science, ol 1681. Sp inge , Be lin, Heidelbe g. doi: 10.1007/3-540-46805-6_19.. [36] L.A. Ga ys, A.S. Ecke , M. Be hge, Image s yle ans e using con olu ional neu al ne wo ks, IEEE Con . on Compu e Vision and Pa e n Recogni ion (2016) 2414–2423. [37] G. An ipo , M. Baccouche, J. Dugelay, Face aging wi h condi ional gene a i e ad e sa ial ne wo ks, IEEE In . Con . in Image P ocessing (2017) 2089–2093. [38] A. Uca , Y. Demi , C. Guzelis, Objec ecogni ion and de ec ion wi h deep lea ning o au onomous d i ing applica ions, Simula ion 93 (2017). [39] J. Redmon, S. Di ala, R. Gi shick, A. Fa hadi, You only look once: Uni ied, eal- ime objec de ec ion, in: IEEE In . Con . in Image P ocessing, 2016. [40] K. Po da , C. Pai, S. Akolka , A con olu ional neu al ne wo k based li e objec ecogni ion sys em as blind aid, 2018.. [41] B. Ma, X. Li, Y. Xia, Y. Zhang, Au onomous deep lea ning: A gene ic DCNN designe o image classi ica ion, Neu ocompu ing 379 (2020) 152–161. [42] Y. Zhang, Y. Wang, X.-Y. Liu, S. Mi, M.-L. Zhang, La ge-scale mul i-label classi ica ion using unknown s eaming images, Pa e n Recogn. 99 (2020) 107100. [43] M. Talo, U.B. Baloglu, O. Yildi im, U.R. Acha ya, Applica ion o deep ans e lea ning o au oma ed b ain abno mali y classi ica ion using MR images, Cogn. Sys . Res. 54 (2019) 176–188. [44] U. Ragha end a, H. Fuji a, S.V. Bhanda y, A. Gudiga , J.H. Tan, U.R. Acha ya, Deep con olu ional neu al ne wo k o accu a e diagnosis o glaucoma using digi al undus images, In . Sci. 441 (2018) 41–49. [45] O. Yildi im, M. Talo, B. Ay, U.B. Baloglu, G. Aydin, U.R. Acha ya, Au oma ed de ec ion o diabe ic subjec using p e- ained 2D-CNN models wi h equency spec um images ex ac ed om hea a e signals, Compu . Biol. Med. 113 (2019) 103387. [46] Wang, L.; Wong, A. COVID-NET: A ailo ed deep con olu ional neu al ne wo k design o de ec ion o COVID-19 cases om ches X- ay images. 2020.. [47] F. Shan, Y. Gao, J. Wang, W. Shi, N. Shi, M. Han, Z. Xue, D. Shen, Y. Shi, Lung in ec ion quan i ica ion o COVID-19 in CT images wi h deep lea ning, 2020.. [48] W. Ouyang, X. Wang, Join deep lea ning o pedes ian de ec ion, IEEE In . Con . in Compu e Vision, 2013. [49] R. G zeszick, J.M. Lenk, F.M. Rueda, G.A. Fink, S. Feldho , M. en Hompel, Deep neu al ne wo k based human ac i i y ecogni ion o he o de picking p ocess, iWOAR 2017.. [50] H. Jiang, E. Lea ned-Mille , Face de ec ion wi h Fas e RCNN, IEEE In . Con . Au oma ic Face Ges u e Recogni ion (2017) 650–657. [51] X. Sun, P. Wu, S.C. Hoi, Face de ec ion using deep lea ning: An imp o ed Fas e RCNN app oach, Neu ocompu ing 299 (2018) 42–50. [52] R. Ranjan, V.M. Pa el, R. Chellappa, A deep py amid de o mable pa model o ace de ec ion. CoRR 2015, abs/1508.04389.. [53] S. Yang, P. Luo, C.C. Loy, X. Tang, Faceness-ne : Face de ec ion h ough deep acial pa esponses, IEEE T ans. Pa e n Anal. Mach. In ell. (2017). [54] W. Wang, J. Yang, J. Xiao, S. Li, D. Zhou, Face ecogni ion based on deep lea ning, in: Q. Zu, B. Hu, N. Gu, S. Seng (Eds.), Human Cen e ed Compu ing, Sp inge , Cham, 2015, pp. 812–820. [55] M. Islam, N. Tasnim, J.-H. Baek, Human gende classi ica ion using ans e lea ning ia Pa e o on ie CNN ne wo ks, In en ions 5 (2020) 16. [56] N. Dalal, B. T iggs, His og am o o ien ed g adien s o human de ec ion, IEEE Con . Comp. Vision and Pa e n Recogni ion (2005) 886–893. [57] C. Co es, V.N. Vapnik, Suppo - ec o ne wo ks, Mach. Lea n. 20 (3) (1995) 273–297. [60] V. Kazemi, J. Sulli an, One millisecond ace alignmen wi h an ensemble o eg ession ees, IEEE Con . Compu e Vision and Pa e n Recogni ion (2014) 1867–1874. [61] J. De Lope, M. G aña, Compa ison o labeling me hods o beha io al ac i i y classi ica ion based on gaze e hog ams, in: E.A. De la Cal, J.R. Villa Flecha, H. Quin ian, E. Co chado (Eds.), Hyb id A i icial In elligen Sys ems, Sp inge , Cham, 2020, pp. 132–144. [62] K. Simonyan, A. Zisse man, Ve y deep con olu ional ne wo ks o la ge-scale image ecogni ion. a Xi 2014, 1409.1556.. [63] S. Gi i, B. Joshi, T ans e lea ning based image isualiza ion using CNN, In . J. A i . In ell. Appl. 10 (4) (2019) 47–55. J. de Lope and M. G aña Neu ocompu ing 500 (2022) 518–527 526