Exploring the balance between interpretability and performance with carefully designed constrainable Neural Additive Models

In o ma ion Fusion 99 (2023) 101882 A ailable online 12 June 2023 1566-2535/© 2023 The Au ho (s). Published by Else ie B.V. This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/). Con en s lis s a ailable a ScienceDi ec In o ma ion Fusion jou nal homepage: www.else ie .com/loca e/in us Full leng h a icle Explo ing he balance be ween in e p e abili y and pe o mance wi h ca e ully designed cons ainable Neu al Addi i e Models E o e Ma io ia,∗, José Ma ía Alonso Mo ala, Albe Ga b aCen o Singula de In es igacion en Tecnoloxias In elixen es (CiTIUS), Uni e sidade de San iago de Compos ela, San iago de Compos ela, Spain bU ech Uni e si y, U ech , The Ne he lands ARTICLE INFO Keywo ds: Gene alised addi i e models Explainable A i icial In elligence In e p e able modelling Neu al addi i e models In e p e abili y Explainabili y ABSTRACT The in e p e abili y o an in elligen model au oma ically de i ed om da a is a p ope y ha can be ac ed upon wi h a se o s uc u al cons ain s ha such a model should adhe e o. O en hese a e in con as wi h he ask objec i e and i is no s aigh o wa d how o explo e he balance be ween model in e p e abili y and pe o mance. In o de o allow an in e es ed use o join ly op imise pe o mance and in e p e abili y, we p opose a new o mula ion o Neu al Addi i e Models (NAM) which can be subjec o a numbe o cons ain s. Acco dingly, ou app oach p oduces a new model ha is called Cons ainable NAM (o jus CNAM in sho ) and i allows he speci ica ion o di e en egula isa ion e ms. CNAM is di e en iable and is buil in such a way ha i can be ini ialised as a solu ion o an e icien ee-based GAM sol e (e.g., Explainable Boos ing Machines). F om his local op imum he model can hen explo e solu ions wi h di e en in e p e abili y-pe o mance adeo s acco ding o di e en de ini ions o bo h in e p e abili y and pe o mance. We empi ically benchma k he model on 56 da ase s agains 12 models and obse e ha on a e age he p oposed CNAM model anks on he Pa e o on o op imal solu ions, i.e., models gene a ed by CNAM exhibi a good balance be ween in e p e abili y and pe o mance. Mo eo e , we p o ide wo illus a i e examples which a e aimed o show s ep by s ep how CNAM wo ks well o sol ing classi ica ion asks, bu also how i can yield insigh s when conside ing eg ession asks. 1. In oduc ion Today’s socie y is accumula ing mo e and mo e da a whose use can op imise exis ing p ocesses and b ing weal h and knowledge. This mo i a es a need o ools ha can p ocess hese da a acco ding o he speci ic asks a hand. Machine Lea ning (ML) and A i icial In elligence (AI) ecen ly gained lo s o success hanks o g ea e pe o mance wi h espec o adi ional app oaches on many benchma k p oblems [1,2]. The ML and AI app oach consis s in speci ying he end goal and cons ain s o be sa is ied and he p og amme au oma ically adjus i sel o sa is y hose. Un o una ely hese models, while e ec i e, a e ypically no unde s andable by humans. This is because hei main job is only o maximise p edic i e gene alisa ion, wi hou aking in o accoun he in elligibili y o he model i sel . The lack o in e p e abili y o some AI-based sys ems is an issue o a numbe o applica ions o which quali y assu ance, us , legal liabili y and adhe ence o e hical p inciples is a mus . Mo i a ed by his, a lo o wo k has been done in he ield o eXplainable A i icial In elligence (XAI) [3,4], a collec i e e o o explain he beha iou ∗Co esponding au ho . E-mail add ess: [email p o ec ed] (E. Ma io i). and p edic ions o ML models o allow debugging, o e sigh , audi ing, knowledge disco e y, and sa e democ a isa ion o hese new powe ul in o ma ion-p ocessing echnologies in many domains [5,6]. In his pape we de ine explainabili y as he abili y o a sys em o gi e an explana ion o he use , ha is a epo o (pa o ) he causal easoning ha lead o a pa icula ou come. We e e o in e p e abili y ins ead o he less demanding p ope y o a sys em o being inspec able in i s pa s in a meaning ul way. In his sense, in e p e abili y is a p e equisi e o explainabili y. XAI has e ol ed in di e en di ec ions based on he equi emen s s a ed by he s akeholde s o he explainable sys ems. Fo example, in o de o build ools ha a e as widely applicable as possible some esea che s ha e p io i ised he de elopmen o ‘‘black-box explain- e s’’ [7]. In his con ex , no assump ion is made on he model and we a e only le wi h he inpu –ou pu ela ionship. In ha case we ha e o ely on pos -hoc app oaches. These ypically in ol e app oxima ing he o iginal model in an app op ia e neighbou hood o da a wi h an in e p e able model (a so-called whi e-box) and hen inspec ing ha o p o ide explana ions, as sugges ed by [8,9] ( his app oach is also h ps://doi.o g/10.1016/j.in us.2023.101882 Recei ed 17 Oc obe 2022; Recei ed in e ised o m 2 May 2023; Accep ed 6 June 2023 In o ma ion Fusion 99 (2023) 101882 2 E. Ma io i e al. known as su oga ing). The su oga ion pa adigm is a e y powe ul one as i allows in p inciple o explain any black-box model, e en deep neu al ne wo ks, as long as he su oga ing models a e powe ul enough o ac like hem. In i ue o his, i we wan o enhance he explana ion o black boxes, we i s need good in e p e able models by design so ha we can la e use hem o su oga ing. To pu i in o he e ms: •we a e in e es ed in explaining black box models as hese a e widely used; •in e p e abili y is a p e equisi e o explainabili y, by he a gu- men s a ed abo e; •we should aim o models which a e in e p e able because hese make ideal su oga e models o explaining o he models. This opens up he ques ion o wha i eally means o a model o be in e p e able. While wi h an impe a i e-s yle app oach (whe e he p og amme is ha d-coded by a human) he in e p e abili y is kep in each sub-componen , wi h he ML-s yle app oach (as i is usually employed) he only solu ions explo ed a e hose ha sol e he ask ega dless o he inal o m o he model. A way o le e aging he powe o he ML app oach is o guide he sea ch o models no only based on hei pe o mance bu also on some o he a ge s ha e lec hei in e p e abili y. In his sense we can imagine ha an app op ia e egula isa ion e m in o he a ge loss could cons ain he op imisa ion o bo h dimensions (i.e., in e p e abili y and pe o mance). Jus as he app op ia e measu e o p edic i e pe o mance a ies om ask o ask, he app op ia e cons ain s o in e ep e abili y a e also applica ion-dependen (usually penalise complexi y and a ou spa se ep esen a ions) and should i e a i ely be e ined wi h domain expe s [10]. Gene alised Addi i e Models (GAM) and Neu al Ad- di i e Models (NAM) a e well-s udied classes o models ha ha e powe ul p edic i e pe o mance while e aining an in e p e able s uc- u e. This wo k add esses a new o mula ion o i ing GAM, named Cons ainable Neu al Addi i e Model (CNAM), such ha app op ia e in e p e able cons ain s can be en o ced o join ly maximise pe o - mance and in e p e abili y. In o de o benchma k his model agains o he app oaches we use a no el in e p e abili y me ic ha is called SHAP-Leng h [11], which exploi s he well-known SHapley Addi i e exPlana ions (SHAP) i s in oduced by [9]. The main con ibu ions in his wo k a e as ollows: •a no el o mula ion o NAM, CNAM, such ha in e p e ablil y cons ain s can be en o ced and join ly op imised alongside ask- ela ed pe o mance me ics; •an ex ensi e benchma k o CNAM on 56 bina y classi ica ion da ase s agains 13 di e en models; •an illus a i e use-case o CNAM on a classi ica ion da ase ; •an illus a i e use-case o CNAM on a eg ession da ase . The es o he manusc ip is o ganised as ollows. In Sec ion 2, we discuss ela ed wo k in he ield. In Sec ion 3, we desc ibe he s uc u e and p ope ies o CNAM. In Sec ion 4, we expe imen i s wi h a bench- ma k s udy o explici ly explo ing CNAM in e p e abili y-pe o mance adeo on bina y classi ica ion asks and hen go deepe wi h wo illus a i e use cases: an as ophysics classi ica ion ask and a socio- economic eg ession ask. Finally in Sec ion 6, we d aw conclusions and delinea e u u e wo k. 2. Rela ed wo k The his o y o lea ning om da a can be aced back o ea ly wo k in he 19 h cen u y whe e, mo i a ed by he desi e o p edic ing as onomical da a and minimising econs uc ion e o s, heo e ical ounda ions and closed- o m solu ions o Linea Models (LM) we e de- eloped, which minimised he Mean Squa ed E o (MSE) as p oposed by [12,13]. The a ge 𝑦is modelled as 𝑦 om a se o ea u es 𝑥𝑖and he ask is o ind a se o eal coe icien s 𝛽𝑖such ha 𝑦 =∑ 𝑖 𝛽𝑖𝑥𝑖 and he MSE= 1∕𝑁∑𝑁 𝑗(𝑦𝑗−𝑦𝑗)2, is minimised. Much la e , [14] uni ied some sca e ed iews o modelling a ce ain a ge wi h a LM and in oduced Gene alised Linea Models (GLM), whe e he a ge 𝑦is ans o med wi h he so-called link unc ion 𝑔(⋅) wi h s a ic coe icien s 𝛼𝑖such ha 𝑔(𝑦) = ∑ 𝑖 𝛼𝑖𝑥𝑖 This new modelling allows (among o he hings) o ha e linea classi ie s by choosing he logis ic sigmoid as he link unc ion, leading o wha is known as Logis ic Reg ession (LR). A u he gene alisa ion o LM was de eloped a Bell Labs by [15, 16] whe e he condi ion o ha ing a s a ic coe icien 𝛼𝑖is elaxed and he ela ionship is allowed o be a non-linea unc ion 𝑓𝑖(⋅)o he uni a ia e ea u e such ha 𝑔(𝑦) = ∑ 𝑖 𝑓𝑖(𝑥𝑖) This new o mula ion is appealing because i is mo e exp essi e han GLM while keeping a ela i ely simple and unde s andable s uc u e. When he model is i ed one can indeed easily isualise each 𝑓𝑖 (also called a shape unc ion) as a unc ion o he 𝑥𝑖 alues, na u ally p o iding bo h a global iew o he beha iou o he model on a da ase and a local explana ion o he p edic ion o a single da a poin . F om a p ac ical poin o iew, ini ially he 𝑓𝑖we e based on Reg ession Splines o deg ee 𝑑o he o m 𝑓𝑖(𝑥𝑖) ∶= ∑𝑑 𝑘=1 𝛽𝑘𝑏𝑗(𝑥𝑖) (Spline-GAM) o o he Ke nel Expansions o he ea u e 𝑥𝑖. On he one hand, his allowed he injec ion o expe knowledge when designing he model. On he o he hand, he i ing p ocedu e was slow and some imes did no con e ge p ope ly. These i ing me hods we e hen shown o be ou pe o med by [17] wi h a p ocedu e o bagging and boos ing bina y decision ees leading o a class o models called Explainable Boos ing Machines (EBM). Fo op imisa ion o speed and memo y EBMs comp ess he ep esen a ion o he unc ion 𝑓𝑖wi h a lookup able, a da a s uc u e ha bins he ea u e alues and maps each bin o he heigh o he shape unc ion. EBMs we e hen u he gene alised o allow also pai wise in e ac- ions (EB2M) o he o m 𝑓(𝑥𝑖, 𝑥𝑗)by [18] and p o ed o be o alue wi h an applica ion in heal hca e [19]. A pa allel de elopmen has a emp ed o model he shape unc ions wi h neu al ne wo ks. [20] pionee ed he wo k wi h he so-called Gene alised Addi i e Neu al Ne wo ks (GANN), whe e each 𝑓𝑖was ep esen ed as a small neu al ne wo k. The op imisa ion p ocess ol- lowed an i e a i e app oach and did no make use o backp opaga ion. GANN we e success ully used o example wi h he aim o imp o ing he pe o mance o c edi sco ing applica ions [21]. Recen ly [22] p oposed Neu al Addi i e Models (NAM), a mode n ein e p e a ion o GANN wi h mo e neu ons, a new ac i a ion unc ion ExU and a sophis ica ed aining p ocedu e ha included d opou , weigh decay, ou pu penal y and ea u e d opou . They epo ed compe i i e ask pe o mance compa ed o EBMs and o he models on ou da ase s. I is wo h no ing ha NAM can be a building block o o he models, o example as he backbone o au oencode s. The main d awback o NAM is he need o a ca e ul se up o many hype pa ame e s and he long aining ime ha is equi ed o he con e gence o hese models. Mo eo e wi h hei o mula ion i is mo e di icul o in oduce expe knowledge in he sys em and o explici ly equi e a pa o a shape unc ion o ha e a speci ic o m. In an e o o pu he human in he loop, [23] buil GAM-Change , a amewo k whe e i ed GAM can be inspec ed and changed when deemed app op ia e. This enables a domain expe o in e ac wi h he sys em and gi e some ools on how o co ec he model whe e ew In o ma ion Fusion 99 (2023) 101882 3 E. Ma io i e al. Table 1 Sho summa y o ela ed models in he li e a u e. CNAM is he new model p oposed in his wo k. LM LR GLM Spline-GAM EBM EB2M NAM CNAM* Classi ica ion asks x x x x x x x Reg ession asks x x x x x x x Can be backbone o Au oencode s x x x Can use p io knowledge on a ge x x x x x x Can use p io knowledge on da a x x Pai wise in e ac ions x A bi a y ini ialisa ion x x x x x Cons ainable x x x Explici ly balancing x in e p e abili y-pe o mance Fig. 1. Schema ic iew o he s uc u e o CNAM ( o classi ica ion) and how cons ain s a e inse ed. Fo eg ession he model is he same wi h he di e ence o no ha ing a logis ic sigmoid non-linea i y a he end. s a is ics a e a ailable bu human knowledge can be o help. While he use does ecei e eedback on he pe o mance o he modi ied model, he e is cu en ly no way o ine- uning he changed model keeping ixed he human in e en ions. Wi h he aim o o e coming d awbacks o cu en s a e o he a models, in his wo k we in oduce he new CNAM, a way o building GAM based on neu al ne wo ks bu wi h a speci ic s uc u e such ha i is possible o ini ialise each shape unc ion as an a bi a y cu e and o en o ce cons ain s on pa s o he ne wo k. This is impo an because i allows us o: 1. le e age he powe and speed o EBM which quickly con e ges o a good solu ion; 2. le e age he exp essi eness o di e en iable p og amming, opening he possibili y o join ly op imising pe o mance and in e p e abili y- ela ed me ics. To conclude his sec ion, Table 1 summa ises he main p ope ies o he mos ou s anding ela ed models in he li e a u e. As we will see in he es o he pape , CNAM is a p omising ool o human-in- he-loop solu ions and can in p inciple be in eg a ed in o GAM-Change enabling a comple e eedback-loop be ween a human and a machine. 3. Model s uc u e We can g aphically ep esen CNAM as illus a ed in Fig. 1. The impo an no el y o CNAM is he cons uc ion o he shape unc ions 𝑓𝑖as a sum o ca e ully ini ialised di e en iable e sions o he uni s ep, which we will call Di e en iable S ep (DS). I is wo h no ing ha only a ew shape unc ions a e expec ed o ake non-ze o alues, hus e ec i ely disca ding some ea u es om he inal sco e by Regula isa- ion (Few Fea u es Selec ed). The pa ame isa ion o he DS is wha will allow CNAM o be ini ialised o any a bi a y shape, in pa icula his p ope y will be used o ini ialise he model o he solu ion o EBM (mo e on his in Sec ion 3.2). Once ini ialised, he s uc u e o CNAM is anspa en enough ha each componen has a clea in e p e a ion and his acili a es he o mal de ini ion o app op ia e egula ise s o he p oblem a hand. As in oduced in Sec ion 1, ha ing a model ha is lexible enough o en o ce di e en cons ain s (e.g., spa si y) is key o a ge ing speci ic in e p e abili y needs o di e en use cases. An enume a ion o di e en desi able cons ain s will be he objec i e o he ollowing sec ion. 3.1. Desi able cons ain s We now p oceed o lis desi able cons ain s ha could be use ul in di e en si ua ions. These can be app op ia e when p io knowledge exis s and can help he model o gene alise be e whe e da a is sca ce: •Mono onici y o he shape unc ions (e.g., in a banking con ex we migh wan o en o ce ha he bigge he ‘‘Loan amoun e m’’ he lowe should be he p obabili y o g an ing he loan). Mo e o mally: o any 𝑥𝑖,𝑥𝑗such ha 𝑥𝑖≤𝑥𝑗we ha e ha 𝑓(𝑥𝑖)≤𝑓(𝑥𝑗)(posi i e mono onici y) o 𝑓(𝑥𝑖)≥𝑓(𝑥𝑗)(nega i e mono onici y). •Smoo hness o he shape unc ions. This could be use ul as a smoo h unc ion is simple o desc ibe compa ed o a jagged one (jagged ep esen a ions a e also mo e likely o be in luenced by noise in he da a). A use migh be in e es ed o explo e i a smoo h ela ionship be ween he ea u e and i s a ge easonably model he ask a hand. Mo eo e , since a smoo h cu e is easie o desc ibe in wo ds han a jagged one, such cons ain becomes hen use ul when he objec i e is o use na u al language in o de o p o ide explana ions [24,25]. In o ma ion Fusion 99 (2023) 101882 4 E. Ma io i e al. Fig. 2. The e ec o changing he pa ame e s o he di e en iable s eps and how hey can o m a 𝑓𝑖. •Few shape unc ions (use ul o p oblems whe e he numbe o ea u es is so la ge ha would be imp ac ical o manually assess e e y shape unc ion). In o he wo ds his en ails ha ing a spa se ep esen a ion ollowing he p inciple o Ockam’s azo : ‘‘i wo models desc ibe he same e en , he model wi h less assump ions (i.e., he model ha elies on ewe a iables) is p e e able’’. I is possible o ma hema ically en o ce some spa seness o a linea combina ion o e ms 𝑥pa ame ised by he do p oduc wi h 𝑤 esul ing in 𝑤.𝑥 by minimising he 𝐿1No m o 𝑤, ha is: 𝑚𝑖𝑛 ∑𝑖|𝑤𝑖| •Local eezing some pa o he shape unc ion (e.g., o help op imise ai ness measu es). Fo example i we had a ca ego ical a iable desc ibing some p o ec ed a ibu e one migh wan o en o ce ha he sco e o he p o ec ed a ibu e ma ch some o mal desc ip ion o ai ness. I is impo an o no e ha he desi able cons ain s lis ed abo e a e no an exhaus i e lis and a e con ex -dependen . Fu he mo e, some o hese cons ain s may con lic wi h maximising ask pe o mance o e en among hemsel es. Fo ins ance, while en o cing mono onici y o a shape unc ion may imp o e he model’s in e p e abili y, i may nega i ely impac i s ask pe o mance. Simila ly, en o cing spa si y o shape unc ions may help wi h in e p e abili y, bu a he cos o some loss in p edic i e accu acy. Ano he app oach ha could be conside ed is o measu e he ag ee- men o ea u e impo ances as indica ed by a human expe wi h wha he model cu en ly p edic s. While his could help he model quickly con e ge o a well-gene alising solu ion, i should be kep in mind ha human anking could in oduce biases ha a e no p esen in he da a. De eloping a sui able me ic o measu ing such ag eemen is a opic o u u e esea ch. Despi e hese conside a ions, he main mo i a ion behind his no el app oach is ha once he desi able cons ain s a e de ined, he op i- misa ion p ocedu e can explo e di e en solu ions ha aim o sa is y all he gi en cons ain s. In he ollowing sec ion, we will discuss how o p ecisely de ine each componen o CNAM and how o de ine app op ia e egula ise s ha en o ce he desi able cons ain s. 3.2. Model cons uc ion CNAM speci ies he s uc u e o how each shape unc ion can be cons uc ed in a a he s aigh o wa d way. In ui i ely, we de elop he idea ha each shape 𝑓𝑖is a sum o 𝑗s ep unc ions, an app oxima ion o indica o unc ions 𝑓𝑖(𝑥𝑖) = ∑ 𝑗 𝑠𝑡𝑒𝑝𝑗(𝑥𝑖) This way he model can be ew i en as 𝑔(𝑦) = ∑ 𝑖∑ 𝑗 𝑠𝑡𝑒𝑝𝑗(𝑥𝑖) As in oduced p e iously, we wan 𝑠𝑡𝑒𝑝(⋅)( he DS) o be a so e - sion o an indica o unc ion ha adds o sub ac s a speci ic (lea nable) quan i y a a ce ain posi ion in he shape unc ion. We pa ame ise DS wi h h ee pa ame e s: 𝑝𝑜𝑠𝑥,𝑝𝑜𝑠𝑦,𝑠𝑝𝑒𝑒𝑑. DS can hen be de ined as ollows: 𝑠𝑡𝑒𝑝(𝑥𝑖) ∶= 𝑝𝑜𝑠𝑦∗𝜎(𝑠𝑝𝑒𝑒𝑑 ∗ (𝑥𝑖−𝑝𝑜𝑠𝑥)) whe e •𝜎(⋅)is he Logis ic Sigmoid unc ion, ha is: 𝜎(𝑥) = 1 1+𝑒−𝑥; •𝑝𝑜𝑠𝑥is he posi ion on 𝑥whe e he shape is cen ed (see Fig. 2(a)); •𝑝𝑜𝑠𝑦is he maximum heigh o he s ep (see Fig. 2(b)); •𝑠𝑝𝑒𝑒𝑑 is a pa ame e ha speci ies how quickly he s ep g ows (see Fig. 2(c)). In o ma ion Fusion 99 (2023) 101882 5 E. Ma io i e al. When many ca e ully ini ialised DS a e added up hey can o m a cu e o a bi a y shape (see Fig. 2(d)). Then, wi hou loss o gene ali y, we can agg ega e all he shape unc ions 𝑓𝑖wi h a linea laye pa ame ised by a ec o 𝛼 o p oduce he inal p edic ion (log-odds in case o classi ica ion), as ollows: 𝑔(𝑦) = ∑ 𝑖 𝛼𝑖⋅𝑓𝑖(𝑥𝑖) The p e ious o mula ion can be use ul o en o cing some egula i- sa ion like he 𝐿1no m on 𝛼𝑖in o de o a ou a spa se ep esen a ion (i.e., when ew shape unc ions a e desi ed). This o mula ion is well-sui ed o bina y classi ica ion asks and eg ession asks. Howe e , we acknowledge ha ex ending ou model o o he asks equi es addi ional conside a ions. One possible app oach o non-bina y classi ica ion is o use a one- s- es ashion, whe e a sepa a e model is ained o each class. This app oach inc eases he complexi y o he sys em, bu i can be e ec i e o mul iclass classi ica ion p oblems. 3.3. En o ceable cons ain s Wi h his o mula ion we can now begin o impose some o he cons ain s discussed in Sec ion 3.1 by speci ying addi ional loss e ms o be join ly op imised o by imposing some s uc u al changes. He e’s how he lis o possible en o ceable cons ain s enume a ed in Sec- ion 3.1 can be implemen ed: •Mono onici y o he shape unc ions: This can be achie ed by imposing 𝑝𝑜𝑠𝑦and 𝑠𝑝𝑒𝑒𝑑 o be always g ea e (o smalle ) han ze o. This can be done by applying he unc ion 𝑚𝑎𝑥(𝑥, 0), also known as ReLU [26,27], o hem. •Smoo hness o he shape unc ions: This can be achie ed by incen i ising small alues o he 𝑠𝑝𝑒𝑒𝑑 pa ame e s o he a ious DS o a gi en shape unc ion. This can be done by minimising he 𝐿2no m o he 𝑠𝑝𝑒𝑒𝑑, ha is o add o he inal loss unc ion he componen 𝜆√∑𝑖𝑠𝑝𝑒𝑒𝑑2 𝑖. The pa ame e 𝜆con ols he amoun o egula isa ion. No ice ha wi h his o mula ion we can ha e smoo h unc ions ha a e s ill able o model big jumps i neces- sa y, some hing ha spline-based GAM ailed o achie e because he splines implici ly encode a p io o global smoo hness. On he o he hand, en o cing an 𝐿2on he 𝑠𝑝𝑒𝑒𝑑 s ill allows local speeds o be e y high i app op ia e. •Few shape unc ions: his can be en o ced by imposing spa si y (i.e., 𝐿1 egula isa ion) on he inal linea laye ha agg ega es he shape unc ions 𝑓𝑖. Tha is, we can ep esen wi hou loss o gene ali y he models as 𝑔(𝑥) = 𝛼1∗𝑓1(𝑥1) + ⋯+𝛼𝑝∗𝑓𝑝(𝑥𝑝) and hen incen i ise a spa se ep esen a ion by adding o he inal loss he 𝐿1no m o 𝛼:𝜆2∑𝑝 𝑖|𝛼𝑖|, whe e 𝜆2con ols he amoun o egula isa ion. This is simila o wha [28] p oposed wi h Lasso, bu i is now possible o apply i o close- o-op imal EBM solu ions, some hing ha up o now was no possible o do. •Local eezing: we can exclude pa s o he model om he op imisa ion p ocess by se ing he g adien o a ce ain s ep o 0. This p e en any u he upda es on ha pa ame e and hus any change on ha pa o he unc ion. 3.4. Pa ame e ini ialisa ion As al eady an icipa ed ea lie , one o he key ea u es o CNAM is he abili y o be ini ialised as solu ions o as e sol e s (e.g. EBM) easily. This is impo an because i we y o op imise he model om a ypical andom ini ialisa ion (e.g., ollowing he p inciples delinea ed by [29]) we obse ed empi ically ha he aining can be uns able and esul in subop imal solu ions wi h espec o EBM. On he o he hand i we op imise he model while s a ing om he solu ion ound by EBM, hen CNAM is able o end up inding solu ions ha ha e simila Fig. 3. Example o applying algo i hm 1 o some andom dummy da a wi h he aim o ini ialising CNAM as a good app oxima ion o EBM. o highe pe o mance while also op imising he in e p e abili y con- s ain s ( hus inding a solu ion ha exhibi s a be e balance be ween pe o mance and in e p e abili y). This will be shown la e wi h he expe imen s in Sec ion 4. Fo he speci ic ini ialisa ion we se he ini ial alue o speed o a la ge alue (we ound 100 o be heu isically a good guess) o be e app oxima e he uni s ep. The ansla ion pa ame e 𝑝𝑜𝑠𝑥can be se as he 𝑥-posi ions o he lookup able o he shape unc ion o EBM. The heigh o he shape unc ion (desc ibed by 𝑝𝑜𝑠𝑦) is inally se o he 𝑦-posi ion o he EBM by i s ini ialising all he 𝑝𝑜𝑠𝑦 o 0and hen adjus ing he 𝑝𝑜𝑠𝑦 om he igh -mos o he le -mos using Algo i hm 1. An illus a ion o how his algo i hm wo ks can be seen in Fig. 3. Tha said, he andom ini ialisa ion can s ill be use ul o some use-cases whe e i is no s aigh o wa d o ain an EBM (e.g. as componen s o an Au oencode ). We s ill sugges o ini ialise he pa- ame e 𝑝𝑜𝑠𝑥 o shape 𝑖as he quan iles o he uni a ia e dis ibu ion o ea u e 𝑖. On he o he hand o classi ica ion and eg ession pu - poses we consis en ly ound in ou p elimina y expe imen s ha EBM al eady con e ges o a close- o-op imal solu ion and ini ialising CNAM o ha as ly imp o es he inal model ac oss bo h pe o mance and in elligibili y me ics. Algo i hm 1 Ini ialise CNAM as EBM Requi e: 𝑦𝐸𝐵𝑀 𝑦𝐶𝑁𝐴𝑀 ←𝑦𝐸𝐵𝑀 ∗ 0 ⊳Ini ialise all as ze o 𝑦𝐶𝑁𝐴𝑀 [−1] ←𝑦𝐸𝐵𝑀 [−1] o 𝑘= 2, 𝑘 ≤𝐿𝑒𝑛𝑔𝑡ℎ(𝑦𝐸𝐵𝑀 )+1, 𝑘 + + do 𝑦𝐶𝑁𝐴𝑀 [−𝑘]←𝑦𝐸𝐵𝑀 [−𝑘]⊳Se he heigh o he desi ed one 𝑦𝐶𝑁𝐴𝑀 [−𝑘+ 1] ←𝑦𝐶𝑁𝐴𝑀 [−𝑘+ 1] − 𝑦𝐶𝑁𝐴𝑀 [−𝑘]⊳ ix he e ec on he ollowing bin end o e u n 𝑦𝐶𝑁𝐴𝑀 3.5. Op imisa ion Once we ha e ini ialised he CNAM pa ame e s, we can op i- mise he inal loss, which includes bo h ask-pe o mance and ask- in e p e abili y e ms, using a g adien -based op imise such as ADAM In o ma ion Fusion 99 (2023) 101882 6 E. Ma io i e al. Fig. 4. A benchma k o he pe o mance-in e p e abili y adeo s ega ding di e en models on 56 da ase s. The 𝑥-axis ep esen s he ela i e Rank o in e p e abili y (measu ed as SHAP-Leng h) while he 𝑦-axis co esponds o he ela i e Rank o pe o mance (measu ed as ROC AUC, Accu acy, F1-Sco e). The bes models lie in he lowe -le pa o he plo s. The do ed blue lines highligh he Pa e o on , i.e. he se o non-domina ed solu ions. Va ious ins an ia ions o CNAM on a e age lie on ha Pa e o on . In e es ingly, e en wi hou egula isa ion, CNAM sco es be e han EBM (wi hou pai wise in e ac ions). Fig. 5. A box plo o he esul s o he benchma k o he di e en models ac oss he di e en s a is ics. In o ma ion Fusion 99 (2023) 101882 7 E. Ma io i e al. Fig. 6. Pe o mance-In e p e abili y adeo on he MAGIC Telescopes da ase . CNAM con igu a ions a e colou ed by he amoun o 𝜆1 egula isa ion, wi h blue indica ing high egula isa ion and g een ep esen ing low egula isa ion. Mo e in e p e able models wi h lowe SL (x-axis) and be e pe o mance (y-axis) a e si ua ed in he uppe - le co ne o he igu e. I is impo an o no e ha 𝜆2also in luences he esul s, bu we ha e omi ed i s isualisa ion o he sake o cla i y. [30]. This echnique allows us o i e a i ely adjus he pa ame e s o CNAM owa ds a solu ion ha minimises he a ge loss, e en wi hou an explici closed- o m o mula o i . To con ol o o e i ing, we pe iodically e alua e agains a alida ion se and employ an ea ly s opping c i e ion [31]. 4. Expe imen s CNAM can be used bo h o bina y classi ica ion o eg ession. We i s benchma k he model wi h di e en egula isa ion pa ame e s on 56 bina y classi ica ion da ase s (see Sec ion 4.1). Subsequen ly, we pe o m a deep di e in o wo speci ic da ase s, a classi ica ion ask (see Sec ion 4.2) and a eg ession ask (see Sec ion 4.3), o p o ide a mo e in-dep h analysis o he model’s beha iou . We implemen ed CNAM as an open-sou ce so wa e (a ailable online a h ps://gi lab. nl4xai.eu/e o e.ma io i/cnam), using Py hon and py o ch [32] as he deep lea ning backend in as uc u e, s uc u ing he code using py o ch-ligh ning lib a y. In ou expe imen s, we explo e he adeo be ween pe o mance and in e p e abili y by a ying he weigh s o he weigh ed loss unc- ion acco ding o he speci ic applica ion domain. Al hough we lack heo e ical con e gence gua an ees, ou empi ical obse a ions sugges ha by ying di e en coe icien s, we can ob ain a di e se se o solu- ions ha e ec i ely na iga e he pe o mance-in e p e abili y adeo . La e in he subsec ions, we demons a e his p ac ically wi h di e en choices o egula isa ion pa ame e s as he added e m o he loss. 4.1. Bina y classi ica ion benchma k In his sec ion we ocus ou a en ion o he ask o bina y classi- ica ion and we benchma k CNAM agains o he classi ie s ac oss 56 abula da ase s aken om [33]. We measu e bo h ask-pe o mance and ask-in e p e abili y sco es. The da ase s a e ela ed o di e en domains and include bo h eal-da a as well as simula ed-da a and spans bo h in size and class imbalance. The basic p ope ies o each da ase a e epo ed in Table 2. In o de o compa e he esul s we ank he models o each da ase so ing hem wi h espec o he speci ic me ic alue hey sco ed. Once his anking is ob ained, we a e age he ank ac oss all he da ase s. We selec ed h ee measu es o classi ica ion pe o mance: •Accu acy: he ac ion (pe cen age) o co ec ly classi ied in- s ances. •F1-Sco e: he geome ic mean o p ecision and ecall. •ROC AUC: he a ea unde he ecei e ope a ing cha ac e is ic cu e (i.e., unde he ROC cu e), ha is he cu e o ue posi i es e sus alse posi i es a all classi ica ion h esholds. Fo measu ing in e p e abili y in e ms o model complexi y we op ed o he so-called SHAP-leng h me ic [11] which is a model- agnos ic me ic which allows us o compa e he e ogeneous models. This me ic e u ns he numbe o SHAP a ibu ions o each da a poin such ha he se o a ibu ions 𝜙𝑖cap u es a gi en ac ion (we se led a 90% he e) o he o e all explana ion mass ∑𝑖|𝜙𝑖|. Mo e o mally: 𝑆𝐿90% ∶= he smalles 𝑖such ha ∑𝑖so ed(|𝜙𝑖|) ∑𝑖|𝜙𝑖|≤0.9. This in ui i ely cap u es he leng h o he ‘‘comp essed’’ explana ion o each p edic ion, ha is wha a use would ha e o ead in o de o ge a ough sense o wha is he impac o di e en ea u es. These leng hs a e hen a e aged ac oss all he da a poin s o he da ase in o de o e u n a single in e p e abili y sco e pe da ase -model pai . I is impo an o no e ha SL migh no always be he ideal measu e o in e p e abili y. Indeed, he bes in e p e abili y me ic depends on he con ex , use p e e ence, and he speci ic domain o he ask. A model wi h a lowe SL will ypically ha e mo e spa se explana ions, so he lowe he SL, he less cogni i e load bu den on he use . This is a key conside a ion when choosing an in e p e abili y me ic, as he goal is o p o ide use s wi h concise ye meaning ul explana ions ha help hem unde s and he model’s beha iou and decision-making p ocess. As s a ed be o e, gi en he sco es o each model on each da ase we can ank each model on he same da ase acco ding o he a ious me ics. Once we ha e compu ed he anking o all he me ics o all he models ac oss all he da ase s we can a e age hem and delinea e he Pa e o on wi h he bes solu ions. The Pa e o on is he se o all he Pa e o-e icien solu ions, ha is all hose solu ions in mul i- objec i e p oblems whe e no o he solu ion is be e han hem in one o he objec i es. We es ed CNAM agains a pool o 12 di e en models, mos o hem implemen ed by he sciki -lea n package [34]: Explainable Boos ing Machines wi h pa iwise in e ac ions (EB2M) and wi hou in e ac ions (EBM-GAM), 100 (Random Fo es wi h 100 ees), 1k (Random Fo es wi h 1000 ees), s m (Suppo Vec o Machine), XGB (eX- emely G adien Boos ing ees), l (Logis ic Reg ession), small- ee (a decision ee wi h maximum ee dep h o 4), big- ee (a decision ee uncons ained), 3-nnc, 5-nnc, 10-nnc ( espec i ely 3-, 5-, and 10- nea es neighbou s classi ie s). In o de o e alua e he capabili ies o CNAM we e alua ed 3 di e - en a ia ions o he p oposed model, each wi h di e en egula isa ion coe icien s. The egula isa ion was se up such ha a spa si y con- s ain (𝐿1no m on he 𝛼) is added o he classical c oss en opy loss: 𝑙𝑜𝑠𝑠 =𝑐𝑟𝑜𝑠𝑠𝑒𝑛𝑡𝑟𝑜𝑝𝑦 +𝜆1∗∑𝑖|𝛼𝑖|. This way 𝜆1becomes a mac oscopic hype pa ame e ha a ou s model simplici y a he po en ial cos o ask pe o mance. We benchma ked CNAM wi h 𝜆1= 0,𝜆1= 0.1, 𝜆1= 1 in o de o show how CNAM beha es wi h no egula isaion, small egula isa ion and s ong egula isa ion. E e y model is e alua ed wi h 5- old s a i ied c oss alida ion. Fo each single da ase he models a e o de ed om he bes o he wo s hus p oducing a anking (lowe alues hus co espond o be e models). The anking is hen a e aged ac oss all he da ase s and he esul s a e epo ed in Figs. 4 and 5. Fig. 4 epo s o he sake o cla i y and eadabili y only he mean o he compu ed ankings. In Fig. 5 he in e es ed eade can see u he de ails in he o m o box plo s ha help unde s and wha is he dis ibu ion o he ankings. The esul s sugges ha CNAM uncons ained is equi alen o (i some imes ma ginally be e han) EBM-GAM. This makes sense as by design CNAM is ini ialised as a good app oxima ion o EBM-GAM. In o ma ion Fusion 99 (2023) 101882 8 E. Ma io i e al. Table 2 Basic p ope ies o each da ase es ed in ou benchma k s udy. Da ase # Ins ances # Fea u es Class imbalance analca da a aids 50 4 0.000000 analca da a asbes os 83 3 0.011758 analca da a bank up cy 50 6 0.000000 analca da a boxing1 120 3 0.090000 analca da a boxing2 132 3 0.005739 analca da a c edi sco e 100 6 0.211600 analca da a cyyoung8092 97 10 0.255181 analca da a cyyoung9302 92 10 0.344518 analca da a aud 42 11 0.145125 analca da a japansol en 52 9 0.001479 analca da a lawsui 264 4 0.732840 appendici is 106 7 0.364543 aus alian 690 14 0.012132 backache 180 32 0.521605 biomed 209 8 0.079691 b eas 699 10 0.096375 b eas cance 286 9 0.164507 b eas cance wisconsin 569 30 0.064940 b eas w 699 9 0.096375 buggyC x 690 15 0.012132 bupa 345 5 0.000412 clean1 476 168 0.016966 cle e 303 13 0.007940 colic 368 22 0.068053 co al 160 6 0.015625 c edi a 690 15 0.012132 c x 690 15 0.012132 ge man 1000 20 0.160000 glass2 163 9 0.004554 hea c 303 13 0.007940 hea h 294 13 0.077792 hea s a log 270 13 0.012346 house o es 84 435 16 0.051795 hunga ian 294 13 0.077792 i ish 500 5 0.012544 labou 57 16 0.088950 lupus 87 3 0.038182 molecula biology p omo e s 106 57 0.000000 monk1 556 6 0.000000 monk2 601 6 0.098895 monk3 554 6 0.001577 mux6 128 6 0.000000 pa i y5 32 5 0.000000 pima 768 8 0.091254 p nn c abs 200 7 0.000000 p nn syn h 250 2 0.000000 p o b 672 9 0.111111 sahea 462 9 0.094470 sona 208 60 0.004530 spec 267 22 0.345762 spec 349 44 0.207560 h eeO 9 512 9 0.004944 ic ac oe 958 9 0.094181 o e 435 16 0.051795 wdbc 569 30 0.064940 xd6 973 9 0.114332 Ins ead, when a egula isa ion o he o m discussed abo e is applied, we can see how he di e en models ac ually loose classi ica ion pe - o mance bu gain in in e p e abili y, e u ning models on he Pa e o on o he bes adeo s. This is impo an because i is an indica ion ha he model canno only e ain compe i i e ask pe o mance bu can success ully explo e he adeo be ween compe ing cons ain s in a way ha he esul lie on he se o Pa e o-e icien solu ions. Mo eo e , CNAM eg 1 u ns up as he second bes om he poin o iew o in e p e abili y (see he plo ela ed o ‘‘Explana ion Leng h’’ on he bo om igh side o Fig. 5), only behind small- ee while i is much be e om he pe o mance iewpoin (i.e., CNAM eg 1 is always be e anked han small- ee in he o he pic u es in Fig. 5). Finally, i is wo h no ing ha in a speci ic applica ion a use should expe imen wi h di e en egula isa ion pa ame e s (and also speci ic egula isa ions i o he speci ic domain) acco ding o his o he needs. 4.2. Illus a i e example on how o add ess a classi ica ion ask: MAGIC da ase In o de o gain mo e in ui ion o he beha iou o CNAM on a speci ic da ase we p o ide as an applica ion he use o CNAM o he MAGIC Telescopes classi ica ion da ase [35], a eal-wo ld da ase ( aken om he UCI eposi o y [36]) ha is much la ge han all o hose included in he se o he p e ious benchma k s udy. In his da ase he ask is o dis inguish be ween wo di e en kind o pa icles (gamma-like s had ons) gi en a ep esen a ion o he image cap u ed by he elescope. F om a physical poin o iew, a pa icle en e s he a mosphe e and p oduce an ellip ic lash o ligh ha is cap u ed by he came a o he elescope. The image is hen desc ibed by he ollowing 10 nume ical ea u es, some o which desc ibe an ellipsoidal shape i ed on he pixels: In o ma ion Fusion 99 (2023) 101882 9 E. Ma io i e al. Fig. 7. Visual ep esen a ion o each shape unc ion o CNAM on he MAGIC Telescopes da ase . Each shape unc ion demons a es he con ibu ion o a gi en ea u e alue o he inal p edic ion, which is he sum o he pa ial sco es. The blue solid line ep esen s he mos in e p e able CNAM ins ance (CNAM bes SL), iden i ied as ha ing he sho es Shap Leng h (SL), while he do ed ed line indica es he bes -pe o ming ins ance (CNAM bes ROC AUC), which has he highes AUC (A ea Unde The Cu e) ROC (Recei e Ope a ing Cha ac e is ics) cu e sco e among models ained wi h a ying lambda1 and lambda2 egula isa ion pa ame e s. The da a dis ibu ion is isualised as a his og am in he backg ound, wi h he coun s epo ed on he igh -axis o he plo . • Leng h: majo axis o ellipse • Wid h: mino axis o ellipse • Size: log 10 o he sum o con en o all pixels (pho on coun ) • Conc: he a io o sum o wo highes pixels o e Size • Conc1: he a io o highes pixel o e Size • Asym: dis ance om highes pixel o cen e, p ojec ed on o majo axis • M3Long: 3 d oo o hi d momen along majo axis • M3T ans: 3 d oo o hi d momen along mino axis • Alpha: angle o majo axis wi h ec o o o igin • Dis : dis ance om o igin o cen e o ellipse The da ase has 19020 ins ances (12332 gamma and 6688 had on), wi h a class imbalance o 0.542. In his ask he Accu acy me ic is no meaning ul as classi ying a had on (backg ound) as a gamma (signal) is wo se han ice e sa (classi ying he signal as backg ound). Fo compa ing di e en classi ie a mo e sensible me ic is ROC AUC, as i allows he e alua ion o he pe o mances a di e en classi ying h esholds. Fo illus a i e pu poses we will i CNAM wi h di e en egula i- sa ion pa ame e s, namely 𝜆1(encou aging spa si y, ha is he shape unc ions a e mo e o en o 0) and 𝜆2(encou aging smoo h shapes), d awing hem andomly om a log-uni o m dis ibu ion 𝑌= log(𝑋) whe e 𝑋=𝑈(−6,1). Each candida e con igu a ion is e alua ed wi h a 5- old s a i ied c oss- alida ion wi h he ROC AUC sco e (pe o mance) and 𝑆𝐿90% (in e p e abili y). Fo compa ison we will also i o he compe i i e models: EBM-GAM (as i is CNAM ini ialisa ion), EB2M and Random Fo es wi h 1000 ees (as bo h models demons a ed high pe o mances on he benchma k s udy). In Fig. 6 is shown how each model sco es in he pe o mance-in e p e abili y space.