On the black-box explainability of object detection models for safe and trustworthy industrial applications

Resul s in Enginee ing 24 (2024) 103498 A ailable online 26 No embe 2024 2590-1230/© 2024 The Au ho (s). Published by Else ie B.V. This is an open access a icle unde he CC BY-NC license (h p://c ea i ecommons.o g/licenses/by- nc/4.0/). Con en s lis s a ailable a ScienceDi ec Resul s in Enginee ing jou nal homepage: www.sciencedi ec .com/jou nal/ esul s-in-enginee ing Resea ch pape On he black-box explainabili y o objec de ec ion models o sa e and us wo hy indus ial applica ions Alain And es a,b,∗, Ai o Ma inez-Se asa, Ibai Lañaa,b, Ja ie Del Se a,c aTECNALIA, Basque Resea ch and Technology Alliance (BRTA), Mikele egi Pasealekua 2, Donos ia-San Sebas ian, 20009, Spain bUni e si y o Deus o, 20012, Donos ia-San Sebas ián, Spain cUni e si y o he Basque Coun y (UPV/EHU), Bilbao, 48013, Spain A R T I C L E I N F O A B S T R A C T Keywo ds: Explainable A ificial In elligence Sa e A ificial In elligence T us wo hy A ificial In elligence Objec de ec ion Single-s age objec de ec ion Indus ial obo ics In he ealm o human-machine in e ac ion, a ificial in elligence has become a powe ul ool o accele a ing da a modeling asks. Objec de ec ion me hods ha e achie ed ou s anding esul s and a e widely used in c i ical domains like au onomous d i ing and ideo su eillance. Howe e , hei adop ion in high- isk applica ions, whe e e o s may cause se e e consequences, emains limi ed. Explainable A ificial In elligence me hods aim o add ess his issue, bu many exis ing echniques a e model-specific and designed o classifica ion asks, making hem less effec i e o objec de ec ion and difficul o non-specialis s o in e p e . In his wo k we ocus on model-agnos ic explainabili y me hods o objec de ec ion models and p opose D-MFPP, an ex ension o he Mo phological F agmen al Pe u ba ion Py amid (MFPP) echnique based on segmen a ion-based masks o gene a e explana ions. Addi ionally, we in oduce D-Dele ion, a no el me ic combining ai h ulness and localiza ion, adap ed specifically o mee he unique demands o objec de ec o s. We e alua e hese me hods on eal-wo ld indus ial and obo ic da ase s, examining he influence o pa ame e s such as he numbe o masks, model size, and image esolu ion on he quali y o explana ions. Ou expe imen s use single-s age objec de ec ion models applied o wo sa e y-c i ical obo ic en i onmen s: i) a sha ed human- obo wo kspace whe e sa e y is o pa amoun impo ance, and ii) an assembly a ea o ba e y ki s, whe e sa e y is c i ical due o he po en ial o damage among high- isk componen s. Ou findings e ince ha D-Dele ion effec i ely gauges he pe o mance o explana ions when mul iple elemen s o he same class appea in a scene, while D-MFPP p o ides a p omising al e na i e o D-RISE when ewe masks a e used. 1. In oduc ion In ecen yea s, A ificial In elligence (AI) has eme ged as a ans- o ma i e o ce ac oss a ious domains, especially in human-machine in e ac ion, whe e i has enabled significan ad ancemen s in da a- d i en decision-making p ocesses. Among hese ad ances, objec de- ec ion has become a key componen , finding applica ion in c i ical a eas such as au onomous d i ing, secu i y su eillance, indus ial au- oma ion, and obo ics [45,22]. S a e-o - he-a objec de ec ion mod- els, including Fas e -RCNN [28], DETR [6], and he YOLO se ies [37], ha e demons a ed imp essi e pe o mance in iden i ying and localiz- ing objec s wi hin images. Despi e hei success, he adop ion o hese models in highly sensi i e en i onmen s emains limi ed, pa icula ly in domains whe e e o s could esul in se ious consequences such as inju y, equipmen damage, o ope a ional ailu es. One o he p ima y * Co esponding au ho a : TECNALIA, Basque Resea ch and Technology Alliance (BRTA), Mikele egi Pasealekua 2, Donos ia-San Sebas ian, 20009, Spain. E-mail add ess: [email p o ec ed] (Alain And es). easons o his hesi ancy is he black-box na u e o objec de ec o s implemen ed as Deep Lea ning models, which o da e amoun o he ma- jo i y o p oposals in he li e a u e. The in e nal ac i a ions o hese a e no inhe en ly in e p e able, making i challenging o end-use s o us he p edic ions issued by objec de ec o s, especially in high- isk en i- onmen s ope a ing in open-wo ld en i onmen s such as au onomous ehicles and indus ial obo ics. In his con ex , he field o Explainable AI (XAI) [5] aims o en- hance he in e p e abili y o AI sys ems by hei audience and ul ima ely, o enhance he use ’s us in he ou pu o AI-based sys ems. Lea ing aside he ca ego y o anspa en AI models (which a e inhe en ly in e - p e able and do no equi e any explana ions o a use o unde s and how hey wo k), explainabili y me hods in XAI can be b oadly ca ego- ized in o whi e-box and black-box app oaches. Whi e-box XAI me hods equi e access o he in e nal wo kings o he model, such as weigh s, h ps://doi.o g/10.1016/j. ineng.2024.103498 Recei ed 24 Oc obe 2024; Recei ed in e ised o m 14 No embe 2024; Accep ed 21 No embe 2024 Resul s in Enginee ing 24 (2024) 103498 2 A. And es, A. Ma inez-Se as, I. Laña e al. ac i a ions, o g adien s (e.g., G ad-CAM [34]). While hese me hods can p o ide powe ul insigh s, hey a e o en limi ed by hei depen- dence on specific model a chi ec u es, making hem difficul o gene al- ize ac oss diffe en models and less accessible o use s un amilia wi h AI esea ch/ ools. In con as , black-box XAI me hods ea he model as an opaque en i y, p o iding explana ions based solely on he mod- el’s inpu -ou pu beha io wi hou equi ing any access o i s in e nal componen s. Howe e , mos black-box XAI me hods a e designed o classifica ion asks a he han o objec de ec ion [29,19,25,3]. While classifica ion models p oduce a single label pe image, ob- jec de ec ion models mus iden i y and localize mul iple objec s wi hin an image. The e o e, hey need o explain no only he class p edic ion o each de ec ed objec – wha hey de ec – bu also he spa ial ea- soning behind he bounding boxes ha define he objec ’s loca ion – whe e he objec is posi ioned wi hin he image. Balancing hese dual aspec s complica es he explana ion p ocess and equi es mo e sophis- ica ed echniques han hose used o classifica ion asks. In his pape , we add ess he gap in XAI me hods o objec de ec ion by ocusing on model-agnos ic, black-box XAI echniques. We p opose and e alua e no el black-box XAI me hods and XAI me ics ha a e speci - ically ailo ed o objec de ec ion models, wi hou equi ing access o in e nal model de ails. Ou p oposed me hods a e gene alizable o objec de ec ion amewo ks beyond hose u ilized in ou expe imen s. Speci - ically, he con ibu ions o his wo k can be summa ized as ollows: •We o mally define a quan i a i e e alua ion me ic, D-Dele ion, which ex ends he exis ing Dele ion me ic [4,25] p oposed o classi- fica ion asks. This me ic is adap ed o handle he unique challenges o objec de ec ion, including localiza ion (as seen in Fig. 4), which is o u mos impo ance when mul iple ins ances o he same objec appea in he same scene. •By using he simila i y sco e o D-RISE [26], we analyze mul iple mask gene a ion me hods’ pe o mance and in oduce D-MFPP, an ex ension o MFPP [42] o iginally de eloped o classifica ion asks. D-MFPP u ilizes segmen a ion-based mask gene a ion o imp o e ex- plana ions o objec de ec ion models. •We analyze he impac o key pa ame e s, such as image dimensions and he model sizes wi hin he YOLO 8 a chi ec u e u ilized in ou expe imen s, which can significan ly influence he quali y o he e- sul ing explana ions. •Las bu no leas , we acili a e he b oade adop ion o he de eloped echniques o objec de ec ion in eal-wo ld use cases by eleasing he code publicly in a eposi o y: h ps://gi hub .com /aklein1995 /d ise _ dm pp _ddele ion. The emainde o his pape is s uc u ed as ollows: in Sec ion 2, we fi s e iew li e a u e ela ed o XAI o objec de ec ion. In Sec ion 3, we p o ide he necessa y backg ound on objec de ec ion and XAI o amilia ize he eade wi h he key concep s used in he defini ions o D-RISE and Dele ion. Nex , Sec ion 4p esen s he expe imen al se up, in- cluding da ase s, objec de ec ion aining configu a ion, employed XAI me hods, and e alua ion me ics. In his sec ion we also in oduce ou p oposed D-MFPP me hod and D-Dele ion me ic. We discuss ou esul s in Sec ion 5. Finally, Sec ion 6concludes he pape wi h a summa y o ou key findings and di ec ions o u u e esea ch. 2. Rela ed wo k Be o e p oceeding wi h he ma e ials and no el me hods in oduced in his wo k, we fi s pause b iefly a XAI me hods, ocusing on hose used o objec de ec ion asks and pu o p ac ice in indus ial applica- ions: XAI me hods. As s a ed in he in oduc ion, XAI offe s insigh s in o he p ocedu e ollowed by an AI-based sys em o elici hei ou pu s, enabling end-use s o unde s and and e en ually us he decisions ou - pu by he AI-based sys em g ounded on objec i e da a [3]. To da e, he majo i y o XAI me hods a e designed o models lea ned o add ess clas- sifica ion asks. Fo ins ance, CAM-based me hods like G adCAM [34], G adCAM++[7]and In eg a ed G adien s [36] quan i y and a ibu e he pixel-wise impo ance o a gi en inpu acco ding o he g adien s wi h espec a a ge class. Mo eo e , making use o backp opaga ion, LRP [20] calcula es he con ibu ion ha a neu on has wi h neu ons in consecu i e laye s o ge ele ance sco es. In con as , pe u ba ion- based echniques wo k by occluding ce ain pa s o he inpu and ana- lyzing i s impac in he p edic ions. Wi hin his ype o echniques, LIME [29], app oxima es a NN wi h an in e p e able model; SHAP [19]as- signs impo ance alues o each inpu ea u e based on Shapley alues; RISE [25] gene a es saliency maps by p obing he model wi h andomly masked e sions o he inpu image; and MFPP [42] gene a es masks by di iding he inpu image in o mul i-scale supe pixels. None heless, none o hem ha e been explici ly ex ended o objec de ec ion asks –wi h he excep ion o RISE, which has been adap ed o his pu pose– al hough echniques like SHAP can also be u ilized o eg ession p ob- lems. XAI me hods o objec de ec ion models. In ecen imes, a sca ci y o XAI app oaches has been p oposed o suppo he in e p e abili y o complex objec de ec ion models. SODEx [33]is a me hod capable o ex- plaining any objec de ec ion algo i hm using classifica ion explaine s, demons a ing how LIME can be in eg a ed wi hin YOLO 4, a a ian o he YOLO amily o single-s age objec de ec o s. Simila ly, D-RISE [26] ex ends RISE’s mask gene a ion echnique by in oducing a new simila i y sco e ha assesses bo h he localiza ion and classifica ion as- pec s o objec de ec ion models. Mo e ecen ly, D-CLOSE [38] enhances D-RISE by p oducing less noisy explana ions. Along wi h o he me hod- ological imp o emen s, D-CLOSE uses mul iple le els o segmen a ion in he mask gene a ion phase. O he app oaches ocusing on hie a chical masking ha e been p oposed. Conc e ely, GSM-NH [41]e alua es he saliency maps a mul iple le els based on he in o ma ion o p e ious less fine-g ained saliency maps, whe eas BODEM [21] u he ex ends his idea bu ocuses on an ex eme black-box scena io whe e only ob- jec coo dina es a e a ailable. XAI me hods o indus ial applica ions. Al hough XAI is inc easingly impo an in indus ial se ings o ensu e sa e y, eliabili y and com- pliance, he adop ion o XAI o objec de ec ion me hods in indus ial use cases has been limi ed o da e [17,15,9]. The as majo i y o he wo ks ocus ei he on image classifica ion, like [8] ha u ilizes G ad- CAM o in e p e ib a ion signal images in he classifica ion o bea ing aul s; ime-se ies da a, e.g. [35] ha p esen s he implemen a ion and explana ions o a emaining li e es ima o model; o abula da a, as in [31]whe e SHAP is used o in e p e and s udy he influence o soil and clima e ea u es on c op ecommenda ions. Rega ding XAI and ob- jec de ec ion o indus ial applica ions, we can find a ew exempla y s udies ha expose he sho age o eal-wo ld use cases cu en ly no ed in his echnological c oss oads. In [23], a ious objec de ec ion mod- els a e e alua ed o hei effec i eness in de ec ing weld cha ac e is ics in adiog aphy images, wi h an emphasis on explainabili y and deploy- men on edge de ices o assis wo ke s. In he same sense, [32]p o ides a comp ehensi e e iew and analysis o a ious XAI echniques applied o objec de ec ion asks in compu e ized omog aphy imaging o med- ical pu poses. Finally, [14] demons a es how o in eg a e G ad-CAM in o he YOLO a chi ec u e and pe o ms expe imen s in bo h public and p i a e da ase s o ehicle on collision and ea - iew came as. 3. Backg ound We now p oceed by elabo a ing on key concep s needed o p ope ly unde s and he de ails o he p oposed D-MFPP echnique and he D- Dele ion me ic ha lie a he co e o his wo k. Conc e ely, we p o ide undamen als o objec de ec ion models (Sec ion 3.1) and XAI, wi h a ocus on model-agnos ic black-box me hods o explain he p edic ions o objec de ec ion models (Sec ion 3.2). Resul s in Enginee ing 24 (2024) 103498 3 A. And es, A. Ma inez-Se as, I. Laña e al. 3.1. Objec de ec o s Objec de ec o s a e c ucial componen s in compu e ision asks, capable o iden i ying and localizing objec s wi hin an image. They can be b oadly ca ego ized in o single-s age and wo-s age de ec o s. Single-s age de ec o s. They di ec ly p edic bounding boxes and class p obabili ies om inpu images in a single pass. Popula single-s age de ec o s, such as YOLO [37], SSD [18]and Re inaNe [30], ea objec de ec ion as a simple eg ession p oblem, s aigh om image pixels o bounding box coo dina es and class p obabili ies. To his end, hey p oduce a dense g id o bounding box p oposals and class p obabili ies in one s ep. Specifically, YOLO [37] di ides he inpu image in o a g id and p edic s bounding boxes and class p obabili ies o each g id cell. Al hough his efficiency is beneficial o eal- ime applica ions, i o en comes a he cos o accu acy when compa ed o wo-s age de ec o s Two-s age de ec o s. These models, among which Fas e R-CNN [28] can be conside ed o be he mos ep esen a i e one, ollow a mo e com- plex app oach ha di ides he de ec ion p ocess in o wo s ages. In he fi s s age, a Region P oposal Ne wo k (RPN) gene a es a se o candida e objec p oposals (bounding boxes) om he inpu image. In he second s age, hese p oposals a e efined and classified in o diffe en objec ca - ego ies by a second ne wo k. This second s age ypically in ol es a mo e complex ne wo k, such as a con olu ional neu al ne wo k (CNN), which pe o ms classifica ion and u he efinemen o he bounding box co- o dina es. This wo-s ep p ocess boos s accu acy by allowing o a mo e efined ea u e analysis, hough i also slows down p ocessing, making wo-s age de ec o s less sui ed o applica ions ha equi e high-speed pe o mance. Mos de ec o ne wo ks, including Fas e R-CNN and YOLO, p o- duce a la ge numbe o bounding box p oposals which a e subsequen ly efined using confidence h esholding and Non-Maximum Supp ession (NMS) o p oduce a se o finally de ec ed objec s in he image. Each bounding box p oposal 𝑑𝑖can be defined as ollows: 𝐝𝑖=[𝐋𝑖,𝑂 𝑖,𝐏𝑖]=[(𝑥𝑖 1,𝑦 𝑖 1,𝑥 𝑖 2,𝑦 𝑖 2),𝑂 𝑖,(𝑝𝑖 1,…,𝑝 𝑖 𝐶)],(1) whe e 𝐋𝑖defines he bounding box co ne s (𝑥𝑖 1, 𝑦𝑖 1)and (𝑥𝑖 2, 𝑦𝑖 2); 𝑂𝑖∈ [0, 1] e e s o he p obabili y ha bounding box 𝐿𝑖con ains an objec o any class; and 𝐏𝑖is a ec o o p obabili ies (𝑝𝑖 1, … , 𝑝𝑖 𝐶) ep esen - ing he p obabili y ha egion 𝐋𝑖belongs o each o 𝐶classes. Unlike adi ional classifie s, which assign a single class label o an en i e im- age, objec de ec o s mus handle bo h classifica ion and localiza ion simul aneously. This dual ask, p edic ing he class and p ecise loca ion o each objec , inc eases he complexi y o making hese models in e - p e able. 3.2. Explainable A ificial In elligence (XAI) Despi e he g ea pe o mance exhibi ed by objec de ec o s in man- i old applica ions, hei adop ion in isk-sensi i e scena ios is o en hin- de ed by a lack o us and anspa ency by he use making decisions based on he de ec ions issued by hese models. As in oduced p e i- ously, esea ch on XAI p oduce echniques and me hods ha make he beha io and p edic ions o AI models unde s andable o humans wi h- ou sac ificing pe o mance [11]. To his end, mul iple XAI echniques ha e been p oposed, which can be classified in o ou b oad ca ego ies [3]: •Scoop-based echniques ocus on he ex en o he explana ion, p o id- ing ei he local explana ions o specific p edic ions o global expla- na ions o he o e all model beha io . •Complexi y-based me hods conside he complexi y o he model, wi h simple , in e p e able models offe ing in insic in e p e abili y and mo e complex models equi ing pos -hoc explana ions. •Model-based app oaches dis inguish be ween XAI me hods ha a e spe- cific o pa icula ypes o models, and hose ha a e model-agnos ic, capable o being applied o any model dis ega ding he specifics o hei in e nals. •Me hodology-based echniques a e ca ego ized by hei me hodological app oach, such as backp opaga ion-based me hods ha ace inpu influences, o pe u ba ion-based me hods ha al e inpu s o obse e changes in he ou pu o he model. Gi en ha objec de ec o s a e ypically complex neu al ne wo ks, hey all unde he complexi y-based ca ego y, he eby equi ing pos -hoc explainabili y me hods o explain hei decisions. Among he a ious me hodology-based echniques, a ibu ion me hods a e commonly used o es ima e he ele ance o each pixel in an image o he de ec ion ask. A ibu ion me hods a e pa icula ly impo an o objec de ec- ion, whe e bo h localiza ion and classifica ion need o be explained. T adi ional a ibu ion me hods ha e been p ima ily de eloped o image classifie s [1], which p oduce a single ca ego ical ou pu , making hem less sui ed o objec de ec o s. Objec de ec o s, unlike classi- fie s, gene a e mul iple de ec ion ec o s ha encode no only class p obabili ies, bu also localiza ion in o ma ion and addi ional me ics, such as objec ness sco es (see Sec ion 3.1). Fu he mo e, echniques like NMS and confidence h eshold fil e ing, which a e used o efine bounding box p oposals, add complexi ies ha equi e a deepe un- de s anding o he model’s in e nal wo kings, complica ing he use o ce ain XAI me hods, such as g adien -based app oaches. The e o e, we ocus on model-agnos ic black-box XAI app oaches, which a e designed o be a chi ec u e-independen , and do no depend a all on he specifics o he model unde a ge . Among model-agnos ic XAI me hods, pe u ba ion-based app oaches a e commonly used due o hei simplici y and effec i eness in e ealing which pa s o he inpu a e mos influen ial o he model’s p edic ions. Pe u ba ion-based echniques offe a di ec way o assess how changes o he inpu image affec he model’s ou pu . By sys ema ically al e ing o masking pa s o he inpu image (using masks o gene a e pe u bed samples), hese me hods allow in e ing he impo ance o diffe en e- gions based on he model’s inpu -ou pu beha io . The ypical pipeline o pe u ba ion-based XAI me hods can be di- ided in o h ee s ages: (1) Da a P epa a ion, (2) Model Assessmen , and (3) Impo ance Compu a ion. In he Da a P epa a ion s age, masks a e gene a ed and applied o he image o c ea e pe u bed samples. The Model Assessmen s age in ol es passing hese pe u bed images h ough he model o obse e he changes in ou pu . Finally, in he Impo ance Compu a ion s age, he impo ance o each pixel is calcu- la ed by compa ing he model’s ou pu s o he o iginal and pe u bed images. While he Model Assessmen s age emains consis en ac oss me hods, wi h each pe u bed image passed h ough he model, he Im- po ance Compu a ion a ies depending on he XAI app oach used. This can ange om simple echniques like e aining a model (e.g., LIME) o mo e complex app oaches. Since he effec i eness o hese me hods la gely depends on how he pe u bed images a e gene a ed, h ee mask gene a ion algo i hms a e nex desc ibed (Fig. 1): •Sliding Window: This me hod, which is simila o he Occlusion ech- nique p oposed in [43], sys ema ically mo es a window o fixed size ac oss he image and se s he egion wi hin he window o a cons an alue (e.g., ze o) o occlude ha pa o he image. By i e a i ely slid- ing he window ac oss he en i e image, we can assess he impac o each occluded egion on he model’s ou pu . The me hod equi es speci ying he window size, which de e mines he a ea o he image being occluded a each s ep, and he s ide, which se s how much he window mo es be ween i e a ions. •RISE: Randomized Inpu Sampling o Explana ion (RISE) [25]in- ol es sampling 𝑁bina y masks o size ℎ ×𝑤, which a e smalle han he o iginal image size 𝐻×𝑊. Each elemen in he mask is independen ly se o 1 wi h p obabili y 𝑝and o 0 wi h he emaining p obabili y 1 −𝑝. These masks a e hen upsampled o size (ℎ+1)⋅𝐶𝐻×(𝑤+1)⋅𝐶𝑊using bilinea in e pola ion, whe e Resul s in Enginee ing 24 (2024) 103498 4 A. And es, A. Ma inez-Se as, I. Laña e al. Fig. 1. Example o h ee masks gene a ed using Sliding Window ( op), RISE (middle), and MFPP (bo om). MFPP masks a e dependen on he image a he inpu o he model. In his case, we conside a sample om he ba e y assembly da ase de ailed in Sec ion 4. 𝐶𝐻×𝐶𝑊=⌊𝐻∕ℎ⌋×⌊𝑊∕𝑤⌋. The upsampled masks a e c opped o he o iginal image size 𝐻×𝑊wi h uni o mly andom offse s anging om (0, 0) o (𝐶𝐻, 𝐶𝑊). This me hod c ea es a di e se se o masks ha co e diffe en pa s o he image, allowing o a comp ehensi e e alua ion o he impo ance o a ious egions. •MFPP: The so-called Mo phological F agmen al Pe u ba ion Py amid (MFPP) [42]me hod di ides he inpu image in o mul i-scale ag- men s and pe u bs hem andomly. In his sense, i is simila o RISE, bu ins ead o pe u bing elemen s o he gene a ed masks wi h di- mension ℎ ×𝑤, MFPP defines egions acco ding o segmen a ions a diffe en scales. Depending on he numbe o defined agmen s, he egions would be mo e fine-g ained ye mo e ime-consuming. The segmen s a e dependen on each image, equi ing he c ea ion o new masks o e e y image. 4. Ma e ials and me hods This sec ion desc ibes he indus ial obo ics use cases in wha e e s o he da ase s (Sec ion 4.1), objec de ec ion model (Sec ion 4.2), XAI me hods (Sec ion 4.3) and he explana ion quali y me ics (Sec ion 4.4) conside ed in ou wo k. The no el XAI echnique and quali y me ics p oposed in his manusc ip a e also desc ibed in Sec ion 4.3. 4.1. Indus ial obo ics da ase s unde conside a ion The da ase s used in his manusc ip ha e been collec ed du ing he cou se o he ULTIMATE p ojec , h ps://ul ima e -p ojec .eu/, which ea u es wo dis inc eal obo ics use cases [16]. The fi s da ase , om PIAP h ps://piap .lukasiewicz .go .pl/, in ol es a collabo a i e wo kspace whe e a human and a obo ic a m wo k oge he . The sec- ond da ase , p o ided by Robo nik h ps:// obo nik .eu/, ocuses on a ba e y assembly a ea, whe e a obo ic a m assembles componen s o a ba e y ki .1 Da ase 1: Human-Robo Da ase . This da ase consis s o 96 images cap u ed om h ee diffe en came as, as exemplified in Fig. 2, wi h 32 images aken om each came a. The da ase includes wo objec classes: human and g ippe . Impo an ly, each image in his da ase con ains 1While he da ase s con ain a ela i ely small numbe o images, his da a sho age is ypically encoun e ed in eal-wo ld indus ial scena ios subjec o da a a ailabili y cons ain s. Ne e heless, in he use cases unde conside a ions he con ex ual and scene a iabili y is minimal, yielding sho - ailed dis ibu- ions o he objec s o be de ec ed. The e o e, he small da ase s desc ibed in he pape sufficien ly cap u e he ele an ea u es o he specific objec de ec ion asks add essed by he models. Fig. 2. Da ase 1 (Human-Robo collabo a ion): Da a a e cap u ed om came as loca ed in 3 diffe en posi ions. All he images belonging o his da ase con ain he aces blu o p ese e anonymi y. Fig. 3. Da ase 2 (Ba e y Assembly ki ): The se up whe e a obo ic a m would assemble he ki based a bi d-eye iew o he able whe e all componen a e expec ed o be; (le ) a heo e ical se up; ( igh ) an ac ual sample. only a single objec o each class, meaning a maximum o one human and one g ippe pe image. To ensu e a di e se and ep esen a i e sample, we applied ea u e ex ac ion using ResNe [12] o ob ain em- beddings o he en i e da ase . The dimensionali y o hese embeddings was educed using P incipal Componen Analysis (PCA), ollowed by K- means clus e ing (wi h 𝑘 =8clus e s). F om each clus e , ou images we e andomly selec ed, esul ing in a final subse . The da a we e spli in o h ee se s: 72 images o aining (75%), 6 o alida ion (6.25%), and 18 o es ing (18.75%). To main ain consis ency, we applied he same pa i ioning o he da a om each came a. This esul ed in 24 im- ages o aining, 2 o alida ion, and 6 o es ing om each came a. Da ase 2: Ba e y Assembly Da ase . This da ase consis s o 7 images, all cap u ed om a bi d’s-eye ( op-down) iew, showing a obo ic a m assembling a ba e y ki , as shown in Fig. 3. The da ase includes fi e dis inc objec ypes: indi idual ba e y,bms_a,bms_b,ba - e y holde , and unknown objec . In con as o he Human-Robo Da ase , each image in he Ba e y Assembly Da ase may con ain mul- iple objec s o he same class, such as se e al indi idual ba e ies in a single scene. I is wo h no ing ha XAI echniques can be applied o any ype o da a. When applied o aining da a, hey help e eal wha he model has lea ned o ocus on du ing aining. When applied o es da a, hey p o ide insigh in o how well he model gene alizes o new, unseen ex- amples. Fo he Human-Robo Da ase , XAI explana ions we e applied exclusi ely o he es images, allowing us o assess he model’s beha - io on unseen da a. Howe e , o he Ba e y Assembly Da ase , gi en he limi ed numbe o images (only 7), XAI explana ions we e applied o he en i e da ase . 4.2. Objec de ec ion model: YOLO 8 Among he possible objec de ec o models, we selec ed one o he s a e-o - he-a op ions, YOLO 8, due o i s nume ous ad ancemen s o e p e ious e sions and i s obus pe o mance in objec de ec ion asks [37]. YOLO 8 [27] in eg a es a no el combina ion o Fea u e Py a- mid Ne wo k (FPN) and Pa h Agg ega ion Ne wo k (PAN) a chi ec u es, enhancing i s abili y o de ec objec s a a ious scales and esolu ions. Resul s in Enginee ing 24 (2024) 103498 5 A. And es, A. Ma inez-Se as, I. Laña e al. The FPN g adually educes he spa ial esolu ion o he inpu image while inc easing ea u e channels, acili a ing mul i-scale objec de ec- ion. The PAN a chi ec u e u he agg ega es ea u es om diffe en le els h ough skip connec ions, imp o ing he de ec ion o objec s wi h di e se sizes and shapes. Addi ionally, YOLO 8 in oduces an ancho - ee de ec ion mechanism ha di ec ly p edic s he cen e o an objec (ins ead o he offse om a known ancho box), educing he numbe o box p oposals and speeding-up he pos -p ocessing. Fu he mo e, i was ained wi h la ge and mo e di e se da ase s including he pop- ula COCO da ase , imp o ing i s pe o mance ac oss a wide ange o images. YOLO 8 was de eloped and eleased by Ul aly ics, and al hough he model and i s weigh s a e open-sou ce, mos use s a e expec ed o u ilize he Ul aly ics amewo k o i s enhanced usabili y. Howe e , unlike p e ious YOLO eleases whe e he p obabili y o each class pe p edic ed box was accessible, in YOLO 8, he Ul aly ics API ou pu s only he p obabili y o he class wi h he highes confidence in each box.2Consequen ly, by de aul , YOLO 8 ou pu s: 𝐝𝑖=[𝐋𝑖,𝑂 𝑖,𝐶 𝑖]=[(𝑥𝑖 1,𝑦 𝑖 1,𝑥 𝑖 2,𝑦 𝑖 2),𝑂 𝑖,𝐶 𝑖],(2) whe e 𝐋𝑖=(𝑥𝑖 1, 𝑦𝑖 1, 𝑥𝑖 2, 𝑦𝑖 2) ep esen s he coo dina es o he bounding box, 𝑂𝑖deno es he objec ness sco e, and 𝐶𝑖co esponds o he p e- dic ed class label o he objec wi hin he bounding box, which diffe s wi h espec o he ou pu s shown in Exp ession (1). 4.3. Explainabili y me hods We e alua e ou popula me hods o gene a ing isual explana- ions o black-box models: LIME, RISE, D-RISE, and D-MFPP. The fi s wo me hods, LIME and RISE,3we e o iginally de eloped o image clas- sifie s bu can be adap ed o objec de ec o s. Howe e , hey p ima ily ocus on explaining classifica ion aspec s and a e no capable o add ess- ing localiza ion cha ac e is ics. In con as , D-RISE is one o he fi s XAI me hods specifically designed o objec de ec o s, p o iding explana- ions ha encompass bo h classifica ion and localiza ion. Addi ionally, we ex end he exis ing MFPP me hod (o iginally ailo ed o classifie s) in o a e sion sui able o objec de ec ion, which we e e o as D-MFPP. In wha ollows we b iefly desc ibe hem, flowing in o a desc ip ion o he p oposed D-MFPP app oach: •LIME was o iginally designed o explain he p edic ions o any clas- sifie by app oxima ing i locally wi h an in e p e able model. To explain he p edic ion o an inpu image 𝐼, LIME fi s an in e p e able model 𝑔(e.g., a linea model) o app oxima e he beha io o he black-box model 𝑓locally a ound 𝐼. The simila i y be ween he o ig- inal image and he pe u bed samples is measu ed using a ke nel unc ion 𝜋𝐼(𝑧). When image explana ions a e a ge ed, LIME g oups con iguous pixels in o supe pixels based on simila ea u es hey ep- esen . This app oach allows LIME o measu e he impo ance o egions in he image a he han indi idual pixels, making he expla- na ions mo e in e p e able. •As in oduced in he p e ious sec ion, RISE [25]was o iginally de- signed o deep neu al ne wo ks ha ake images as inpu and ou - pu a class p obabili y (e.g., a classifie like ResNe -50). I gene a es saliency maps ha indica e he impo ance o each pixel by applying andomly gene a ed bina y masks 𝑀𝑖 o he inpu image 𝐼and obse - ing he changes in he model’s ou pu 𝑓(𝐼⊙𝑀 𝑖). In RISE, 𝑁bina y masks 𝑀𝑖∈{0, 1}ℎ×𝑤a e gene a ed (as explained in Sec ion 3.2). 2h ps://gi hub .com /ul aly ics /ul aly ics /issues /2863% h ps://gi hub .com /ul aly ics /ul aly ics /issues /4908. 3These XAI me hods ha e been chosen due o hei pe u ba ion-based na u e, which aligns closely wi h he me hodology ollowed by he XAI me hods D- RISE and D-MFPP p oposed in his wo k. Bo h D-RISE and D-MFPP gene a e explana ions h ough pe u ba ions. These masks a e hen applied o he inpu image 𝐼 o gene a e masked images 𝐼′ 𝑖=𝐼⊙𝑀 𝑖, whe e ⊙deno es elemen -wise mul iplica ion. The model is e alua ed on each masked image 𝐼′ 𝑖 o ob ain he ou - pu s 𝑓(𝐼⊙𝑀 𝑖). The impo ance sco e o each pixel (𝑥, 𝑦)is hen calcula ed as he weigh ed sum o he ou pu s: 𝑆𝐼,𝑓 (𝑥, 𝑦)= 1 𝑁 𝑁 ∑ 𝑖=1 𝑓(𝐼⊙𝑀 𝑖)⋅𝑀𝑖(𝑥, 𝑦)(3) whe e he weigh s 𝑀𝑖(𝑥, 𝑦) ep esen he alue o mask 𝑖a pixel (𝑥, 𝑦). The in ui ion behind RISE is ha 𝑓(𝐼⊙𝑀 𝑖)would be high when pix- els p ese ed by mask 𝑀𝑖a e impo an . Al hough his is ue when ha ing infini e di e se masks, in p ac ice RISE calcula es each pix- el’s impo ance empi ically by Mon e Ca lo sampling. The e o e, RISE la gely depends on he numbe o masks (𝑁) and how hey a e gen- e a ed (i.e., is sensi i e o he selec ed p obabili y 𝑝and esolu ion 𝑠). 4.3.1. D-RISE and p oposed D-MFPP app oach Unlike he o he wo app oaches o iginally designed o classifie s ha measu e solely classifica ion aspec s, D-RISE (De ec o Random- ized Inpu Sampling o Explana ion) [26]was designed o explain bo h he classifica ion and localiza ion o a de ec ion. In his sense, D-RISE ex- ends RISE by p oducing saliency maps specifically o objec de ec o s. As p e iously seen in Sec ion 3.1, he ou pu gi en by an objec de ec- o diffe s om he p obabili y ec o gi en by a classifie , ob aining localiza ion in o ma ion 𝐿𝑖, an objec ness sco e 𝑂𝑖and he p obabili y o classi ying each bounding box o any o he conside ed classes 𝑃𝑖. As a consequence, Exp ession (3)used by RISE is eplaced in D-RISE wi h a new simila i y sco e, gi en by: 𝑆𝐼,𝑓 (𝐝𝑡,𝐝𝑗)=𝑠𝐿(𝐝𝑡,𝐝𝑗)⋅𝑠𝑃(𝐝𝑡,𝐝𝑗)⋅𝑠𝑂(𝐝𝑡,𝐝𝑗),(4) whe e 𝑠𝐿=𝐼𝑜𝑈(𝐋𝑡, 𝐋𝑗), 𝑠𝑃=𝐏𝑡⋅𝐏𝑗∕(||𝐏𝑡|| ⋅||𝐏𝑗||), and 𝑠𝑂=𝑂𝑗. In his o mula ion, 𝑠𝐿 ep esen s he spa ial p oximi y o he bounding boxes encoded by he a ge de ec ion 𝐝𝑡and he p oposal 𝐝𝑗, measu ed using he In e sec ion o e Union (IoU); he e m 𝑠𝑃e alua es he simila i y be ween he class p obabili ies o he a ge de ec ion and he p oposal using cosine simila i y; and 𝑠𝑂inco po a es he objec ness sco e o he p oposal 𝑂𝑗. I is impo an o no e ha o a de ec ion a ge 𝐝𝑡 he e would po en ially be mo e han one de ec ion p oposals 𝐝𝑗. The e o e, we would ha e mul iple 𝑆𝐼,𝑓(𝐝𝑡, 𝐝𝑗). As explained in D-RISE, he expla- na ions conside only he de ec ion wi h maximal sco e o each mask: 𝑆𝐼,𝑓 (𝐝𝑡,𝑓(𝑀𝑖⊙𝐼))=max 𝐝𝑗∈𝑓(𝑀𝑖⊙𝐼)𝑆𝐼,𝑓(𝐝𝑡,𝐝𝑗).(5) Gi en he YOLO 8 ou pu s explained in Sec ion 4.2, which do no p o ide he class p obabili y ec o 𝑃𝑖wi hou modi ying i s a chi ec- u e (an app oach we wan o a oid wi hin he scope o his pape ), we mus adap he simila i y sco e o only conside 𝑠𝐿and 𝑠𝑂. Conse- quen ly, he modified simila i y sco e can be exp essed as: 𝑆𝐼,𝑓 (𝐝𝑡,𝐝𝑗)=𝑠𝐿(𝐝𝑡,𝐝𝑗)⋅𝑠𝑂(𝐝𝑡,𝐝𝑗)=𝐼𝑜𝑈(𝐋𝑡,𝐋𝑗)⋅𝑂𝑗.(6) This adjus men allows s ill u ilizing D-RISE effec i ely o gene - a ing saliency maps wi h he de aul YOLO 8 model, ocusing on he spa ial and objec ness aspec s o de ec ions, while main aining he in- eg i y o he model’s o iginal a chi ec u e. Simila ly, we can adop his simila i y sco e bu apply i wi h a diffe en mask gene a ion p ocess. The MFPP me hod in oduced in Sec- ion 3.2, o iginally designed o classifica ion asks, can be ex ended by applying Equa ion (6), esul ing in D-MFPP. To he bes o ou knowl- edge, no p e ious wo k has p oposed his a ian o MFPP o objec de ec ion asks. Resul s in Enginee ing 24 (2024) 103498 6 A. And es, A. Ma inez-Se as, I. Laña e al. 4.4. Me ics E alua ing he pe o mance o a ibu ion-based explainabili y me h- ods o image da a in ol es assessing how well he gene a ed ele ance hea maps highligh impo an egions o he inpu image ha con ibu e o he model’s decision. Gene ally, acco ding o [13], explana ion qual- i y me ics can be g ouped in o six ca ego ies based on hei logical simila i y: ai h ulness, obus ness, localiza ion, complexi y, andom- iza ion, and axioma ic me ics. In his s udy, we ocus on wo o hese ca ego ies ha a e pa icula ly ele an o objec de ec ion: localiza ion (Sec ion 4.4.1) and ai h ulness (Sec ion 4.4.2). 4.4.1. Localiza ion Localiza ion me ics e alua e whe he he explainable e idence is cen e ed a ound a egion o in e es (RoI) defined by a bounding box, segmen a ion mask, o a cell wi hin a g id. These me ics aim o e i y i he saliency maps co ec ly highligh he a eas in he image ha con ain he objec o in e es . Among hem, ou expe imen s will conside : •Poin ing Game (PG), which is a human e alua ion me ic in oduced in [44]. I he highes saliency poin lies inside he human-anno a ed bounding box o an objec , i is coun ed as a hi . The PG accu acy is gi en by: PG =#𝐻𝑖𝑡𝑠 #𝐻𝑖𝑡𝑠+#𝑀𝑖𝑠𝑠𝑒𝑠 ,(7) which is a e aged o e all ca ego ies in he da ase . •Ene gy-based Poin ing Game (EBPG) [39], which measu es he p opo - ion o ac i a ions wi hin he gi en bounding box ela i e o he whole ac i a ion in he image. I assesses how much o he model’s ac i a- ion ene gy is concen a ed wi hin he p edefined egion o in e es . Fo mally: EBPG =∑(𝑥,𝑦)∈bbox 𝑆𝐼,𝑓 (𝑥, 𝑦) ∑(𝑥,𝑦)∈bbox 𝑆𝐼,𝑓 (𝑥, 𝑦)+∑(𝑥,𝑦)∉bbox 𝑆𝐼,𝑓(𝑥, 𝑦),(8) whe e 𝑆𝐼,𝑓 (𝑥, 𝑦) ep esen he saliency sco e a pixel (𝑥, 𝑦), ∑(𝑥,𝑦)∈bbox 𝑆𝐼,𝑓 (𝑥, 𝑦) ep esen s he sum o ac i a ion alues wi hin he bounding box, and ∑(𝑥,𝑦)∉bbox 𝑆𝐼,𝑓(𝑥, 𝑦) ep esen s he sum o ac i a ion alues ou side he bounding box. 4.4.2. Fai h ulness Me ics accoun ing o ai h ulness quan i y o wha ex en explana- ions ollow he p edic i e beha io o he model, asse ing ha mo e impo an ea u es play a la ge ole in model ou comes. These me ics ocus on unde s anding he causal ela ionship be ween inpu ea u es and he model’s ou pu by sys ema ically al e ing he ea u es and ob- se ing he changes in p edic ions. Among hem: •Dele ion: Inspi ed by he wo k by [4], he Dele ion me ic was p o- posed in RISE [25]. This me ic measu es a dec ease in he p obabili y o he p edic ed class as mo e and mo e impo an pixels a e emo ed, whe e he impo ance is ob ained om he saliency map. A sha p d op, and hus a low A ea Unde he p obabili y Cu e (AUC, as a unc ion o he ac ion o emo ed pixels), indica es a good explana- ion. Gi en he impo ance sco e o each pixel calcula ed by any XAI me hod, 𝑆𝐼,𝑓, we can o mula e he Dele ion me ic as: Dele ion(I,S,c) =AUC({𝑃𝑟(𝑓(𝐼⊙𝑀 𝑘)=𝑐)}𝐾 𝑘=1),(9) whe e 𝐼is he o iginal image, 𝑀𝑘 ep esen a mask wi h he 𝑘- h mos impo an pixels emo ed so ed by 𝑆𝐼,𝑓 , 𝑃𝑟(𝑓(𝐼⊙𝑀 𝑘)=𝑐) ep esen s he p obabili y o model 𝑓p edic ing ha he bounding box belongs o class 𝑐, and AUC(⋅) compu es he a ea unde he cu e o he 𝐾p edic ions. •Minimum Subse : I ollows he same logic as Dele ion, bu ins ead o de e mining he AUC, i conside s he equi ed numbe o pixels ha make he p edic ion o change [10]. Gi en he impo ance sco e o each pixel (𝑆𝐼,𝑓 ), Min-Subse is defined as he smalles subse o pixels ha needs o be emo ed o change he model’s p edic ion. Ma hema - ically: Min-Subse (𝐼,𝑆,𝑐)= min {𝑘∈{1,2…,𝐾}∶ 𝑓(𝐼⊙𝑀 𝑘)≠𝑓(𝐼)},(10) whe e 𝑓(𝐼⊙𝑀 𝑘) ep esen s he class label assigned by he model 𝑓 a e passing he image 𝐼wi h he op 𝑘mos impo an pixels e- mo ed, and 𝑓(𝐼)is he class label p edic ed o he o iginal image. 4.4.3. P oposed D-dele ion and D-minimal subse me ics O iginally, Dele ion was designed o classifie s. Howe e , wi h ob- jec de ec o s, mul iple de ec ions in a single image can occu . Al hough D-RISE s a ed he necessi y o adap his me ic o objec de ec o s [26], no o mal defini ion can be ound in he li e a u e. The e o e, conside - ing he impo ance o his issue in eal use cases, we o mally e-define Equa ion (9)in wo manne s: 1. Dele ion. Measu es he explana ion gi en he a ge class label 𝐶𝑡( e- ga dless i he e is mo e han one elemen o a class) and i e a i ely emo es he op 𝑘mos impo an pixels: Dele ion(𝐼,𝑆,𝐶𝑡)= 𝐴𝑈𝐶 ⎛⎜⎜⎝{max 𝐝𝑘 𝑗[𝑂𝑘 𝑗 ⋅I{𝐶𝑘 𝑗=𝐶𝑡}]}𝐾 𝑘=1⎞⎟⎟⎠ .(11) The model 𝑓(⋅) akes as inpu he masked image 𝐼⊙𝑀 𝑘and ou - pu s a se o bounding box p oposals 𝐝𝑘 𝑗=[𝐋𝑘 𝑗, 𝑂𝑘 𝑗, 𝐶𝑘 𝑗]. The indica o unc ion I{𝐶𝑘 𝑗=𝐶𝑡}equals 1 i he p edic ed class 𝐶𝑘 𝑗ma ches he a ge class 𝐶𝑡, and 0 o he wise. The e m max𝐝𝑘 𝑗[𝑂𝑘 𝑗 ⋅I{𝐶𝑘 𝑗=𝐶𝑡}] selec s he maximum objec ness sco e 𝑂𝑘 𝑗 o he bounding boxes whe e he p edic ed class ma ches he a ge class. The AUC is hen compu ed o e he se o p edic ion sco es o he 𝐾s eps, whe e a each s ep he mos impo an pixels a e p og essi ely emo ed. 2. D-Dele ion. While he s anda d Dele ion me ic e alua es he impac o pixel emo al on a class p edic ion, i lacks he abili y o accoun o spa ial localiza ion, which is essen ial in objec de ec ion asks whe e mul iple ins ances o he same class can appea . D-Dele ion add esses his limi a ion by ocusing on a specific a ge bounding box 𝐝𝑡, conside ing bo h he class in o ma ion, 𝐶𝑡, and IoU be ween he a ge and o he de ec ed p oposals, 𝐝𝑘 𝑗. This ensu es ha he me ic no only measu es ai h ulness bu also akes localiza ion in o accoun , p o iding mo e p ecise explana ions in si ua ions whe e di - e en objec s o he same class coexis . Ma hema ically is exp essed as: D-Dele ion(𝐼,𝑆,𝐶𝑡)= 𝐴𝑈𝐶 ⎛⎜⎜⎝{max 𝐝𝑘 𝑗[𝑂𝑘 𝑗 ⋅I{𝐶𝑘 𝑗=𝐶𝑡}⋅I{𝐼𝑜𝑈(𝐝𝑡,𝐝𝑘 𝑗)>𝛾}]}𝐾 𝑘=1⎞⎟⎟⎠ (12) whe e 𝛾is a h eshold. As a consequence, when mul iple elemen s o he same class a e in an image, D-Dele ion will only conside hose p oposals 𝐝𝑘 𝑗p edic ed by he model ha ha e a p edefined IoU wi h he a ge bounding box 𝐝𝑡. The diffe ence be ween Dele ion and D-Dele ion is illus a ed in Fig. 4. This figu e highligh s how D-Dele ion dis inguishes be ween diffe en objec s o he same class by inco po a ing localiza ion in o - ma ion, leading o mo e efined and accu a e explana ions (↓AUC in he Figu e’s las ow) when mul iple objec s o he same class a e de- ec ed in an image. Fo he sake o cla i y, we p o ide he pseudocode o Dele ion in Algo i hm 1, whe e he main diffe ence wi h espec o D-Dele ion a e lines 10 o 12. Resul s in Enginee ing 24 (2024) 103498 7 A. And es, A. Ma inez-Se as, I. Laña e al. Fig. 4. Illus a ion o a collabo a i e wo kspace ea u ing wo humans and a obo ic a m. The fi s ow shows he o iginal image. The second ow displays he image wi h he 10% mos impo an pixels emo ed o each human, as iden ified by an XAI me hod. In he hi d ow, he Dele ion me ic cu e, which only conside s class ype, shows a high p obabili y sco e e en when he p ima y human is la gely occluded by he o he pe son. The ou h ow p esen s he D- Dele ion me ic cu e, which inco po a es a localiza ion componen , p o iding a mo e accu a e measu e o explana ion impo ance by conside ing he posi ions o en i ies wi hin he image. A lowe a ea unde he cu e indica es a be e explana ion. Las ly, akin o how Min-Subse is ela ed o Dele ion, D-Min-Subse is associa ed wi h D-Dele ion. Consequen ly, D-Min-Subse conside s bo h he class ype and he IoU o de e mine he numbe o pixels equi ed o make he p edic ion o change: D-Min-Subse (𝐼,𝑆,𝐶𝑡)= Algo i hm 1 Dele ion Me ic’s Pseudocode o Objec De ec o . Requi e: Image 𝐼, saliency map 𝑆𝐼,𝑓 , numbe o s eps 𝐾, a ge de ec ion 𝐝𝑡, a ge class 𝐶𝑡 1: Ini ialize 𝑆←[] 2: o 𝑘 =1 o 𝐾do 3: 𝑀𝑘←𝑆𝐼,𝑓 emo ing he op 𝑘mos impo an pixels 4: Apply mask 𝑀𝑘 o image 𝐼 5: Fo wa d pass h ough he model 𝑓and ob ain he bounding box p opos- als 𝐝𝑗=[𝐋𝑗, 𝑂𝑗, 𝐶𝑗] =𝑓(𝐼⊙𝑀 𝑘) 6: Ini ialize lis o p oposals: p oposals ←[] 7: o each bounding box 𝐝𝑗p edic ed by he model 𝑓do 8: i 𝐶𝑗=𝐶𝑡 hen 9: p oposals.append(𝑂𝑗) % Fo D-Dele ion 10: i 𝐼𝑜𝑈(𝐝𝑡, 𝐝𝑗) >𝛾 hen 11: p oposals.append(𝑂𝑗) 12: end i 13: else 14: p oposals.append(0) 15: end i 16: end o 17: Inse he maximum sco e wi hin he dele ion buffe : 𝑆←𝑆∪ max(p oposals) 18: end o 19: Calcula e he Dele ion me ic as he a ea unde he cu e: 𝐷=AUC(𝑆) 20: e u n Dele ion sco e 𝐷 min {𝑘∈{1,2…,𝐾}∶ 𝐶𝑘 𝑗≠𝐶𝑡o 𝐼𝑜𝑈(𝐝𝑡,𝐝𝑘 𝑗)<𝛾},(13) whe e 𝐶𝑘 𝑗 ep esen s he p edic ed class label o de ec ion 𝑗when pass- ing he masked image 𝐼⊙𝑀 𝑘 h ough he model 𝑓, wi h 𝐝𝑘 𝑗=𝑓(𝐼⊙𝑀 𝑘) being he se o de ec ions a e emo ing he op 𝑘mos impo an pix- els. In his con ex , D-Min-Subse depends on wo condi ions: (1) he class p obabili y labels 𝐶𝑘 𝑗 o he p edic ed bounding box 𝐝𝑘 𝑗mus no longe ma ch he a ge class 𝐶𝑡, o (2) he IoU be ween he a ge bounding box 𝐝𝑡and he p edic ed bounding box 𝐝𝑘 𝑗 alls below he h eshold 𝛾. The minimum 𝑘is iden ified as he s ep whe e ei he o hese condi ions is fi s me . 5. Expe imen s and esul s Con a ily o mos s udies in he XAI li e a u e ha p ima ily o- cus on benchma k da ase s, ou esea ch wo k ocuses on assessing he explainabili y o objec de ec o s in eal-wo ld indus ial da a. In his con ex , o e alua e he effec i eness o explana ions, we o mula e ou key esea ch ques ions o answe hem wi h empi ical e idence: •RQ1: Which XAI me hod p o ides he mos eliable and insigh ul explana ions o objec de ec ion models? •RQ2: Does he D-Dele ion me ic enhance he us wo hiness o XAI ou pu s when mul iple objec s o he same class a e p esen in he image? •RQ3: How does he mask gene a ion p ocess influence he quali y o explana ions, pa icula ly when using simila i y sco es o objec de ec ion? How does D-MFPP beha e? •RQ4: Do diffe en image dimensions impac he explana ions gene - a ed by XAI me hods? Do models o a ying sizes (la ge, medium, small, nano) ocus on diffe en egions o he image in hei explana- ions? Nex , we ou line he hype pa ame e s used ac oss ou expe imen s o ensu e consis ency in aining and e alua ion. Fo bo h da ase s, models we e ained using he YOLO 8 a chi ec u e o a o al o 100 epochs. The image size (imgsize) was se o he la ges dimension o he inpu image (e.g., 720 × 1280 ←←→ 1280), and da a augmen a ion echniques such as andom ho izon al flipping and colo ji e we e applied. Fo consis- ency, he de aul Ul aly ics se ings we e used whe e e applicable. In Resul s in Enginee ing 24 (2024) 103498 8 A. And es, A. Ma inez-Se as, I. Laña e al. Table 1 Quan i a i e me ics o LIME, RISE and D-RISE o he Human-Robo da ase . The able p esen s he pe o mance o each XAI echnique in e ms o classifica ion (Dele ion, D-Dele ion, Min-Subse , D-Min-Subse ) and localiza ion me ics (PG and EBPG), wi h sco es epo ed o each class (Human,G ippe ) and he o e all a e age. Lowe alues a e be e o me ics ma ked wi h ↓, while highe alues a e be e o hose ma ked wi h ↑.Bold alues indica e he bes a e age sco es ac oss all objec s, highligh ing he bes -pe o ming XAI me hod o each me ic. Values highligh ed in g ay ep esen he bes sco es o each objec ca ego y (Human o G ippe ) and should be in e p e ed e ically. XAI Me hod Objec Dele ion (↓) D-Dele ion (↓)Min-Subse (↓) D-Min-Subse (↓)PG(↑)EBPG(↑) LIME Human 0.0759 0.0632 4.2703 4.2703 1.0000 34.984 G ippe 0.4688 0.0324 1.3859 1.3859 1.0000 2.2270 A e age 0.2723 0.0478 2.8281 2.8281 1.0000 18.6060 RISE Human 0.1827 0.1241 9.0108 9.0108 0.7500 19.0824 G ippe 0.2637 0.0060 0.6355 0.63 1.0000 1.0542 A e age 0.2232 0.0651 4.8232 4.8232 0.875 10.0683 D-RISE Human 0.1255 0.0818 5.7335 5.7335 0.8750 20.4061 G ippe 0.2777 0.0059 0.6091 0.6091 1.0000 1.0815 A e age 0.2016 0.0438 3.1713 3.1713 0.9375 10.7438 Fig. 5. Hea maps ob ained by applying RISE (le ) and D-RISE ( igh ) o he de ec ion o a human in he Human-Robo Da ase . he case o LIME, we use he baseline implemen a ion o [29], whe e we adop he SLIC segmen a ion algo i hm [2](wi h 100 segmen s) and gene a ed 1000 samples o assess he quali y o he p oduced explana- ions. Fo RISE and D-RISE, we employed 5000 masks wi h a p obabili y o 0.25 and a esolu ion o 16 ×16 o p oduce he saliency maps. Las ly, o all objec de ec ion p edic ions, a confidence h eshold o 0.7 was se o de e mine he alidi y o each de ec ion. In wha ollows we p esen and discuss on he esul s ob ained o answe each o he RQ o mula ed abo e: RQ1: Compa ison be ween XAI me hods In he Human-Robo da ase , he compa ison be ween LIME, RISE, and D-RISE, as shown in Table 1, e eals dis inc s eng hs ac oss di - e en me ics (Sec ion 4.4). LIME pe o ms good in e ms o localiza- ion, wi h highe PG and EBPG sco es (100% and 18.60%, espec i ely) compa ed o D-RISE (93.75% and 10.74%). This indica es ha LIME gene a es mo e localized saliency maps, ocusing closely on he bound- ing boxes o de ec ed objec s. Howe e , his supe io pe o mance is pa ly due o he size o he objec being analyzed. LIME’s supe pixel gene a ion is be e sui ed o la ge objec s (e.g., human), as la ge e- gions o he image can be g ouped effec i ely in o meaning ul segmen s, leading o highe localiza ion sco es. This ad an age also applies o clas- sifica ion, whe e la ge objec s allow LIME o be e p ese e ele an ea u es o de ec ion. Con e sely, o smalle objec s (e.g., g ippe ), LIME s uggles when compa ed o he o he me hods, as eflec ed by i s wo se pe o mance me ics in hose cases. In con as , RISE and D-RISE a e less sensi i e o objec size, making hem mo e obus ac oss diffe en objec scales, which is e iden in hei be e pe o mance on smalle objec s like he g ippe . They achie e Dele ion sco es o 0.2636 and 0.2777, espec i ely, compa ed o LIME’s 0.4688. When conside ing he o e all pe o mance ac oss classes, RISE, wi h a Dele ion sco e o 0.2232 and D-Dele ion o 0.0651, shows im- p o emen o e LIME in classifica ion- ela ed asks bu s ill lags behind D-RISE, which achie es he lowes Dele ion (0.2016) and D-Dele ion (0.0438) sco es. Al hough D-RISE offe s he bes balance be ween clas- sifica ion and localiza ion, he diffe ence be ween RISE and D-RISE is Fig. 6. Explana ions o a scene using diffe en s ide and window size configu a- ions when using D-Sliding Window (combina ion o mask gene a ion explained in Sec ion 3.2 and Equa ion (6)). (Le ) S ide o 16; (Righ ) S ide o 8; (Top) Window size o 32; (Bo om) Window size o 64. minimal in his da ase , whe e each image con ains only a single objec pe class. As a esul , as shown in Fig. 5, hei hea maps a e e y simi- la o each o he , bo h highligh ing he human head. Howe e , D-RISE elimina es less ele an a eas mo e effec i ely. In he esul s ob ained o e he Ba e y Assembly da ase (Table 2), a simila pa e n can be no iced. LIME excels a localiza ion wi h an a e age EBPG o 16.03%, while RISE and D-RISE pe o m be e in e- aining key classifica ion ea u es. Since his da ase includes mul iple objec s o he same class (e.g., mul iple ba e ies), bo h LIME and RISE, which a e no designed o handle mul iple de ec ions o he same class, expose se e e limi a ions. RISE, wi h a D-Dele ion sco e o 0.1474, p e- se es key ea u es be e han LIME, bu is ou pe o med by D-RISE, which achie es a sco e o 0.0344. D-RISE also shows he highes PG sco e (97%), pe o ming significan ly be e han LIME (76.85%) and RISE (66.95%). O e all, when dealing wi h da ase s con aining only one objec pe class, he diffe ences be ween LIME, RISE, and D-RISE a e ela i ely small in quan i a i e e ms. Howe e , when mul iple objec s o he same class appea in a gi en inpu image, D-RISE clea ly domina es o e he es o echniques. As illus a ed in Fig. 7, D-RISE gene a es cohe en hea maps o each de ec ed objec in he Ba e y Assembly da ase , whe eas LIME and RISE p o ide a global saliency map o he en i e class. By combining he indi idual saliency maps om D-RISE, a mo e accu a e and objec -specific explana ion can be p oduced. This also highligh s he limi a ions o LIME and RISE when applied o mul- Resul s in Enginee ing 24 (2024) 103498 9 A. And es, A. Ma inez-Se as, I. Laña e al. Table 2 Quan i a i e me ics o LIME, RISE and D-RISE o he Ba e y Assembly da ase . The able p esen s he pe o mance o each XAI echnique in e ms o classifica ion (Dele ion, D-Dele ion, Min-Subse , D-Min-Subse ) and localiza ion me ics (PG and EBPG), wi h sco es epo ed o each objec (indi ba , bms a, bms b, unknown objec , ba holde ) and he o e all a e age. Lowe alues a e be e o me ics ma ked wi h ↓, while highe alues a e be e o hose ma ked wi h ↑.Bold alues indica e he bes a e age sco es ac oss all objec s, highligh ing he bes -pe o ming XAI me hod o each me ic. G ay-highligh ed alues ep esen he bes sco es o each objec ca ego y (indi ba , bms a, bms b, unknown objec o ba holde ) and should be in e p e ed e ically. XAI Me hod Objec Dele ion (↓) D-Dele ion (↓)Min-Subse (↓) D-Min-Subse (↓)PG(↑)EBPG(↑) LIME indi ba 0.7549 0.2806 44.3412 14.6756 0.3188 2.5347 bms a 0.0184 0.0181 0.8784 0.8784 1.0000 30.7573 bms b 0.0125 0.0125 0.8321 0.8321 1.0000 16.5440 unknown objec 0.0624 0.0245 2.0342 2.0342 1.0000 20.1071 ba holde 0.1498 0.0849 6.8115 4.3766 0.5238 10.2087 A e age 0.1996 0.0841 10.9795 4.5594 0.7685 16.0304 RISE indi ba 0.7659 0.4359 81.2344 32.0482 0.0144 1.2558 bms a 0.0190 0.0190 1.9417 1.9417 1.0000 3.0409 bms b 0.0217 0.0217 2.1266 2.1266 1.0000 2.4561 unknown objec 0.5333 0.2008 3.8372 3.8372 1.0000 5.5780 ba holde 0.0902 0.0595 7.9519 5.5632 0.3333 3.3681 A e age 0.2860 0.1474 19.4184 9.1034 0.6695 3.1398 D-RISE indi ba 0.6214 0.0311 35.7678 2.7725 1.0000 2.0546 bms a 0.0181 0.0181 1.9880 1.9880 1.0000 3.2955 bms b 0.0128 0.0116 1.3407 1.3407 1.0000 2.8876 unknown objec 0.0879 0.0485 3.7448 4.2071 0.8571 7.6831 ba holde 0.4839 0.0626 21.7291 5.1009 1.0000 4.9635 A e age 0.2448 0.0344 12.9141 3.0819 0.9714 4.1768 iple objec s, as hei global saliency maps do no diffe en ia e be ween indi idual ins ances. RQ2: D-dele ion me ic o scenes wi h mul iple objec s o he same class As a seconda y obse a ion in he expe imen s o RQ1, he D- Dele ion me ic is specifically designed o o e come he limi a ions o adi ional dele ion me ics, pa icula ly when mul iple objec s o he same class a e p esen in an image. In he Ba e y Assembly da ase , whe e se e al ins ances o he same class (e.g., indi ba ) appea , D-Dele ion demons a es clea ad an- ages. By inspec ing Table 2, RISE, while pe o ming easonably well wi h an a e age Dele ion sco e o 0.2860, i s ill ob ains a ela i ely high D-Dele ion sco e o 0.1474, sugges ing ha i s uggles o di - e en ia e be ween he con ibu ions o indi idual objec s. In con as , D-RISE, which ob ains an a e age Dele ion sco e o 0.2448, ou pe o ms RISE wi h a D-Dele ion sco e o 0.03444. This highligh s D-RISE’s abil- i y o isola e and p ese e key ea u es o each objec , p o iding mo e us wo hy, objec -specific explana ions a he han b oad, class-le el insigh s. The Min-Subse and D-Min-Subse me ics, which measu e he min- imal p opo ion o pixels needed o dis up a de ec ion, ein o ce hese findings. In he Human-Robo da ase , Table 1, whe e only one ob- jec pe class appea s, he diffe ences be ween Dele ion and D-Dele ion sco es a e mino , and he Min- Subse and D-Min-Subse alues a e close o each o he . Howe e , in he Ba e y Assembly da ase , whe e he diffe ences be ween Dele ion and D-Dele ion a e mo e subs an ial and mul iple objec s o he same class co-occu in he same image, he Min-Subse (12.9141) and D-Min-Subse (3.0819) alues also di e ge sig- nifican ly. RQ3: Influence o he mask gene a ion s a egy When compa ing XAI app oaches o objec de ec ion asks config- u ed wi h diffe en mask gene a ion echniques, he esul s in Tables 3 and 4ini ially sugges ha D-Sliding Window pe o ms he bes in al- mos all me ics. Howe e , as no ed in he cap ions, his is only in cases whe e explana ions we e p o ided. Fo he Human-Robo da ase (Ta- ble 3), ega dless o he window size and s ide, D-Sliding Window ailed o p o ide explana ions o la ge objec s, such as humans, and only p o ided meaning ul explana ions o smalle objec s like g ippe in- s ances. Simila ly, in he Ba e y Assembly da ase (Table 4), D-Sliding Window s uggled wi h la ge objec s when using a smalle window size (w=32), which led o highe sco es in classifica ion me ics. E en wi h Fig. 7. Hea maps gene a ed in a scene o he Ba e y Assembly da ase o wo a ge classes: indi idual ba e y ( op ow) and ba e y holde (bo om ow). The fi s and second columns shows he saliency maps ob ained using LIME and RISE, independen o he numbe o objec s o he same class in he image. Columns 3,4 and 5 display hea maps gene a ed using D-RISE o h ee diffe en indi idual elemen s o he same class.