Jou nal o Vision (2021) 21(2):2, 1–18 1
The join ole o geome y and illumina ion on ma e ial
ecogni ion
Manuel Lagunas Uni e sidad de Za agoza, I3A, Za agoza, Spain
Ana Se ano Uni e sidad de Za agoza, I3A, Max Planck Ins i u e o
In o ma ics, Za agoza, Spain
Diego Gu ie ez Uni e sidad de Za agoza, I3A, Za agoza, Spain
Belen Masia Uni e sidad de Za agoza, I3A, Za agoza, Spain
Obse ing and ecognizing ma e ials is a undamen al
pa o ou daily li e. Unde ypical iewing condi ions,
we a e capable o effo lessly iden i ying he objec s
ha su ound us and ecognizing he ma e ials hey a e
made o . Ne e heless, unde s anding he unde lying
pe cep ual p ocesses ha ake place o accu a ely
disce n he isual p ope ies o an objec is a
long-s anding p oblem. In his wo k, we pe o m a
comp ehensi e and sys ema ic analysis o how he
in e play o geome y, illumina ion, and hei spa ial
equencies affec s human pe o mance on ma e ial
ecogni ion asks. We ca y ou la ge-scale beha io al
expe imen s whe e pa icipan s a e asked o ecognize
diffe en e e ence ma e ials among a pool o candida e
samples. In he diffe en expe imen s, we ca e ully
sample he in o ma ion in he equency domain o he
s imuli. F om ou analysis, we find significan fi s -o de
in e ac ions be ween he geome y and he
illumina ion, o bo h he e e ence and he candida es.
In addi ion, we obse e ha simple image s a is ics and
highe -o de image his og ams do no co ela e wi h
human pe o mance. The e o e, we pe o m a high-le el
compa ison o highly nonlinea s a is ics by aining a
deep neu al ne wo k on ma e ial ecogni ion asks. Ou
esul s show ha such models can accu a ely classi y
ma e ials, which sugges s ha hey a e capable o
defining a meaning ul ep esen a ion o ma e ial
appea ance om labeled p oximal image da a. Las , we
find p elimina y e idence ha hese highly nonlinea
models and humans may use simila high-le el ac o s
o ma e ial ecogni ion asks.
In oduc ion
Unde ypical iewing condi ions, humans a e
capable o e o lessly ecognizing ma e ials and
in e ing many o hei key physical p ope ies, jus
by b ie ly looking a hem. Al hough his is almos
an e o less p ocess, i is no a i ial ask. The
image ha is inpu o ou isual sys em esul s om
a complex combina ion o he su ace geome y, he
e lec ance o he ma e ial, he dis ibu ion o ligh s in
he en i onmen , and he obse e ’s poin o iew. To
ecognize he ma e ial o a su ace while being in a ian
o o he ac o s o he scene, ou isual sys em ca ies
ou an unde lying pe cep ual p ocess ha is no ye
ully unde s ood (Adelson, 2000;D o e al., 2001a;
Fleming e al., 2001).
So how does ou b ain ecognize ma e ials? We could
hink ha , simila o sol ing an in e se op ics p oblem,
ou b ain is es ima ing he physical p ope ies o each
ma e ial (Pizlo, 2001). This would imply knowledge
o many o he physical quan i ies abou he objec
and i s su ounding scene, om which ou b ain could
disen angle he e lec ance o he su ace. Howe e ,
we a ely ha e access o such p ecise in o ma ion,
so a ia ions based on Bayesian in e ence ha e been
p oposed (Ke s en e al., 2004).
O he app oaches a e based on image s a is ics,
and explain ma e ial ecogni ion as a p ocess whe e
ou b ain ex ac s image ea u es ha a e ele an o
desc ibe ma e ials. Then, i would y o ma ch hem
wi h p e iously acqui ed knowledge, o disce n he
ma e ial we a e obse ing. Conside ing his app oach
ou isual sys em would dis ega d he illumina ion,
mo ion, o o he ac o s in he scene and y o
ecognize ma e ials by ep esen ing hei ypical
appea ance in e ms o ea u es ins ead o explici ly
acqui ing an accu a e physical desc ip ion o each
ac o . This ype o image analysis can be ca ied ou
in he p ima y domain (Adelson, 2008;Fleming, 2014;
Geisle , 2008;Mo oyoshi e al., 2007;Nishida & Shinya,
1998), o in he equency domain (B ady & Oli a,
Ci a ion: Lagunas, M., Se ano, A., Gu ie ez, D., & Masia, B. (2021). The join ole o geome y and illumina ion on ma e ial
ecogni ion. Jou nal o Vision,21(2):2, 1–18, h ps://doi.o g/10.1167/jo .21.2.2.
h ps://doi.o g/10.1167/jo .21.2.2 Recei ed May 4, 2020; published Feb ua y 3, 2021 ISSN 1534-7362 Copy igh 2021 The Au ho s
This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 In e na ional License.
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 2
Figu e 1. Two sphe es made o sil e , unde wo diffe en
illumina ions, leading o comple ely diffe en pixel-le el
s a is ics.
Figu e 2. Two objec s o diffe en geome ies bu made o he
same ma e ial, unde he same illumina ion. The objec on he
le seems o be made o a shinie ma e ial.
2012;Giesel & Zaidi, 2013;Oli a & To alba, 2001).
Howe e , i is a gued i ou isual sys em ac ually
de i es any aspec s o ma e ial pe cep ion om such
simple s a is ics (Ande son & Kim, 2009). Fo ins ance,
Fleming and S o s (2019) ha e ecen ly p oposed he
idea ha highly nonlinea encodings o he isual inpu
may be e explain he unde lying p ocesses o ma e ial
pe cep ion.
In his wo k, we ho oughly analyze how he
con ounding e ec s o illumina ion and geome y
in luence human pe o mance in ma e ial ecogni ion
asks. The same ma e ial can yield di e en appea ances
owing o changes in illumina ion and/o geome y
(Figu es 1 and 2), al hough i is possible o ha e wo
di e en ma e ials look he same by weaking he wo
pa ame e s (Vango p e al., 2007). We aim o u he
ou unde s anding o he complex in e play be ween
geome y and illumina ion in ma e ial ecogni ion.
We ha e ca ied ou la ge-scale, igo ous online
beha io al expe imen s whe e pa icipan s we e asked
o ecognize di e en ma e ials, gi en images o one
e e ence ma e ial and a pool o candida es. By using
pho o ealis ic compu e g aphics, we ob ain ca e ully
con olled s imuli, wi h a ying deg ees o in o ma ion
in he equency domain. In addi ion, we obse e
ha simple image s a is ics, image his og ams, and
his og ams o V1-like subband il e s do no co ela e
wi h human pe o mance in ma e ial ecogni ion asks.
Inspi ed by Fleming and S o s’ ecen wo k (2019),
we analyze highly nonlinea s a is ics by aining a
deep neu al ne wo k. We obse e ha such s a is ics
de ine a obus and accu a e ep esen a ion o ma e ial
appea ance and ind p elimina y e idence ha hese
models and humans may sha e simila high-le el ac o s
when ecognizing ma e ials.
Ma e ial ecogni ion
Recognizing ma e ials and in e ing hei key ea u es
by sigh is in aluable o many asks. Ou expe ience
sugges s ha humans a e able o co ec ly p edic a
wide a ie y o ough ma e ial ca ego ies like ex iles,
s ones, o me als (Fleming & Bül ho , 2005;Fleming,
2014;Ged e al., 2010;Li & F i z, 2012); o i ems
ha we would call “s u ” (Adelson, 2001)—like sand
o snow. Humans a e also capable o iden i ying he
ma e ials in a pho og aph by b ie ly looking a hem
(Sha an e al., 2009,2008) o o in e ing hei physical
p ope ies wi hou he need o ouch hem (Fleming
e al., 2013,2015a;Ja abo e al., 2014;Maloney &
B aina d, 2010;Nagai e al., 2015;Se ano e al.,
2016). This abili y is buil om expe ience, by ac ually
con i ming isual imp essions wi h o he senses. This
way, ma e ial pe cep ion becomes a cogni i e p ocess
(Palme , 1975) whose unde lying in icacies a e no
ully unde s ood ye (Ande son, 2011;Fleming e al.,
2015b;Thompson e al., 2011).
In e play o geome y and illumina ion
Ma e ial pe cep ion is a complex p ocess ha
in ol es a la ge numbe o dis inc dimensions (Mao e
al., 2019;Obein e al., 2004;Sè e, 1993) ha , some imes,
a e impossible o physically measu e (Hun e e al.,
1937). The illumina ion o a scene (Beck & P azdny,
1981;Bousseau e al., 2011;Zhang e al., 2015)and
he shape o a su ace, a e esponsible o he inal
appea ance o an objec (Nishida & Shinya, 1998;
Schlü e & Faul, 2019;Vango p e al., 2007) and,
he e o e, o ou pe cep ion o he ma e ials i is
made o (Olkkonen & B aina d, 2011). Humans a e
capable o es ima ing he e lec ance p ope ies o
a su ace (D o e al., 2001b) e en when he e is no
in o ma ion abou i s illumina ion (D o e al., 2001a;
Fleming e al., 2001), ye we pe o m be e unde
illumina ions ha ma ch eal-wo ld s a is ics (Fleming
e al., 2003). Indeed, geome y and illumina ion ha e a
join in e ac ion in ou pe cep ion o glossiness (Faul,
2019;Leloup e al., 2010;Ma low e al., 2012;Olkkonen
& B aina d, 2011)andcolo (Bloj e al., 1999). In his
wo k, we explo e he in e play o shape, illumina ion,
and hei spa ial equencies in ou pe o mance a
ecognizing ma e ials. To achie e ha , we launched
igo ous online beha io al expe imen s whe e we ely
on ealis ic compu e g aphics o gene a e he s imuli
and ca e ully a y hei in o ma ion in he equency
domain.
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 3
Image s a is ics and ma e ial pe cep ion
One o he goals in ma e ial pe cep ion esea ch is
o un angle he p ocesses ha happen on ou isual
sys em o comp ehend hei oles and know wha
in o ma ion hey ca y. The e is an ongoing discussion
on whe he ou isual sys em is sol ing an in e se
op ics p oblem (Kawa o e al., 1993;Pizlo, 2001)o i i
ma ches he s a is ics o he inpu o ou isual sys em
(Adelson, 2000;Mo oyoshi e al., 2007;Thompson
e al., 2016) o unde s and he wo ld ha su ounds us.
La e s udies ega ding ou isual sys em and how we
pe cei e ma e ials dismiss he in e se op ics app oach
and claim ha i is unlikely ha ou b ain es ima es
he pa ame e s o he e lec ance o a su ace, when,
o ins ance, we wan o measu e glossiness (Fleming,
2014;Geisle , 2008). Ins ead, hey sugges ha ou
isual sys em joins low and midle el s a is ics o make
judgmen s abou su ace p ope ies (Adelson, 2008).
On his hypo hesis, Mo oyoshi e al. (2007) sugges ha
he human isual sys em could be using some so o
measu e o his og am symme y o dis inguish glossy
su aces. O he wo ks ha e explo ed image s a is ics in
he equency domain (Hawken & Pa ke , 1987;Schille
e al., 1976), o ins ance, o cha ac e ize ma e ial
p ope ies (Giesel & Zaidi, 2013), o o disc imina e
ex u es (Julesz, 1962;Scha ali zky & Zisse man, 2001).
Howe e , i is a gued ha , i ou isual sys em ac ually
de i es any aspec s o ma e ial pe cep ion om simple
s a is ics (Ande son & Kim, 2009;Kim & Ande son,
2010;Olkkonen & B aina d, 2010). Ins ead, ecen
wo k by Fleming and S o s (2019) p oposes ha , o
in e he p ope ies o he scene, ou isual sys em
is doing an e icien and accu a e encoding o he
p oximal s imulus (image inpu o ou isual sys em).
Thus, highly nonlinea models, such as deep neu al
ne wo ks, may be e explain human pe cep ion. In line
wi h such obse a ions, Bell e al. (2015)showhow
deep neu al ne wo ks can be ained in a supe ised
ashion o accu a ely ecognize ma e ials, and Wang
e al. (2016) la e ex end i o also ecognize ma e ials
in ligh ields. Close o ou wo k, Lagunas e al. (2019)
de ise a deep lea ning-based ma e ial simila i y me ic
ha co ela es wi h human pe cep ion. They collec ed
judgemen s on pe cei ed ma e ial simila i y as a whole,
no explici ly aking in o accoun he in luence o
geome y o illumina ion, and build hei me ic upon
such judgemen s. In con as , we ocus on analyzing o
which ex en geome y and illumina ion do in e e e
wi h ou pe cep ion o ma e ial appea ance. We
launch se e al beha io al expe imen s wi h ca e ully
con olled s imuli, and ask pa icipan s o speci y
which ma e ials a e close o a e e ence. In addi ion,
aking inspi a ion om hese ecen wo ks, we explo e
how highly nonlinea models, such as deep neu al
ne wo ks, pe o m in ma e ial classi ica ion asks.
We ind ha such models a e capable o accu a ely
Figu e 3. G aphical use in e ace o he online beha io al
expe imen s. In pa icula , his sc eensho belongs o he TEST
SH. On he le , he use can see he e e ence ma e ial
oge he wi h he cu en selec ion. On he igh , she can
obse e all he candida e ma e ials. To selec one candida e
ma e ial, he use clicks on he co esponding image and i is
au oma ically added o he selec ion box on he le .
ecognizing ma e ials, and u he obse e ha deep
neu al ne wo ks may sha e simila high-le el ac o s o
humans when ecognizing ma e ials.
Me hods
We ca ied ou a se o online beha io al
expe imen s whe e we analyze he in luence o
geome y, illumina ion, and hei equencies in human
pe o mance o ma e ial ecogni ion asks. Pa icipan s
a e p esen ed wi h a e e ence ma e ial and hei main
ask is o pick i e ma e ials om a pool o candida es
ha hey hink a e close o he e e ence. A sc eensho
o he expe imen can be seen in Figu e 3.
S imuli
We ob ain ou s imuli om he da ase p oposed by
Lagunas e al. (2019). This da ase con ains images
c ea ed using pho o ealis ic compu e g aphics, wi h 15
di e en geome ies, 6 di e en eal-wo ld illumina ions
anging om indoo scena ios o u ban o na u al
landscapes, and 100 di e en ma e ials measu ed om
hei eal-wo ld coun e pa s which we e pooled om
Mi subishi Elec ic Resea ch Labo a o ies (MERL)
da abase (Ma usik e al., 2003). We sample he ollowing
ac o s o ou expe imen s:
Geome ies. Among he geome ies ha he da ase
con ains, we choose he sphe e and Ha an-2 geome y
(Ha an e al., 2016). These a e low and high spa ial
equency geome ies, espec i ely, sui able o es how
he spa ial equencies o he geome y a ec he inal
appea ance o he ma e ial and ou pe o mance a
ecognizing i .
•Sphe e: Rep esen ing a smoo h, and low spa ial
equency geome y, widely adop ed in p e ious
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 4
Figu e 4. Examples o he s imuli in each diffe en online
beha io al expe imen . On he le , we show an example o he
e e ence s imuli wi h one o he six illumina ions. On he igh ,
we show a small subse (6 o he 100 ma e ials) o he
candida e s imuli wi h S . Pe e s illumina ion.
beha io al expe imen s (Filip e al., 2008;Ja abo
e al., 2014;Ke & Pellacini, 2010;Sun e al., 2017).
•Ha an-2:1I is a geome y wi h high spa ial
equencies, and wi h high spa ial a ia ions ha
has been ob ained h ough op imiza ion echniques.
•Ha an-2: Su ace has had signi ican success
in ecen pe cep ual s udies and applica ions
(Gua ne a e al., 2018;Guo e al., 2018;Lagunas
e al., 2019;Vá a & Filip, 2016).
The s imuli in each di e en expe imen can be
obse ed in Figu e 4. The geome y in he e e ence
and candida e samples changes depending on he
expe imen , he de ails a e as ollows:
• Tes HH: Bo h he e e ence and he candida es
depic Ha an geome y.
• Tes HS: The e e ence depic s Ha an and he
candida es depic he sphe e.
• Tes SH: The e e ence depic s he sphe e while he
candida es depic Ha an.
• Tes SS: Bo h he e e ence and he candida es
depic he sphe e geome y.
Illumina ions. To p e en a pu e ma ching ask, we
choose di e en illumina ions be ween he e e ence
and candida e ma e ials o all beha io al expe imen s.
• The e e ence samples depic six di e en
illumina ions cap u ed om he eal wo ld. All
illumina ions can be obse ed in Figu e 5.To
ha e an in ui ion o he con en in he cap u ed
illumina ion, he inse s show he RGB in ensi y o
he ho izon al pu ple line. We use all illumina ions
in he da ase since hey con ain a mix o spa ial
equencies sui able o empi ically es how he
spa ial equencies o he illumina ion may a ec
human pe o mance on ma e ial ecogni ion asks.
The illumina ions G ace,Ennis,andU izi ha e
a b oad spa ial equency spec um, Pisa and
Doge mainly con ain medium and low-spa ial
equency con en , while Glacie mainly has
Figu e 5. Le : All illumina ions depic ed in he online beha io al
expe imen s. The inse co esponds o he pixel in ensi y o
he ho izon al pu ple line. Righ : Magni ude spec um o he
luminance o each illumina ion.
low-spa ial equency con en . To simpli y he
no a ion, we e e o hem h oughou he a icle
as high- equency, medium- equency, and
low- equency illumina ions, espec i ely.
• The candida e samples depic he S . Pe e s
illumina ion (excep in an addi ional expe imen
discussed in he Discussion whe e hey depic Doge
illumina ion). S . Pe e s is an illumina ion ha has
been used in he pas o se e al pe cep ual s udies
(Fleming e al., 2003;Se ano e al., 2016), and i
can be seen in Figu e 5. The inse shows he RGB
pixel in ensi y o he ho izon al pu ple line.
To quan i y he spa ial equencies o he
illumina ions, we ha e employed he high- equency
con en (HFC) measu e (B ossie e al., 2004). This
measu e cha ac e izes he equencies in a signal by
summing linea ly weigh ed alues o he spec al
magni ude, hus a oiding o a bi a ily choose a
sepa a ion be ween high and low equencies, o isually
assessing he slope o he 1/ ampli ude spec um. A
high HFC alue means highe equencies in he signal.
Figu e 6 shows he HFC o each illumina ion.
Ma e ials
We use all he ma e ials om he Lagunas e al.
da ase Lagunas e al. (2019). The e e ence ials a e
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 5
Figu e 6. HFC measu e compu ed o all he candida e and
e e ence illumina ions. We can obse e how high- equency
illumina ions (Uffizi,G ace,Ennis,S . Pe e s) also ha e a high
HFC alue, medium- equency illumina ions (Pisa,Doge)ha ea
lowe HFC alue, and, las , low- equency illumina ions (Glacie )
ha e he lowes HFC alue.
sampled uni o mly o co e all 100 ma e ial samples
in he da ase . Examples o he s imuli used in each
beha io al expe imen a e shown in Figu e 4,whe e he
image on he le shows he e e ence ma e ial and he
igh a ea shows a subse o he candida e ma e ials.
Pa icipan s
The online beha io al expe imen s we e designed
o wo k ac oss pla o ms on s anda d web b owse s,
and hey we e conduc ed h ough he Amazon
Mechanical Tu k (MTu k) pla o m. In o al, 847
unique use s ook pa in hem (368 use s belonging
o he expe imen s explained in Resul s, and 479
belonging o he addi ional expe imen s explained
in he Discussion), 44.61% o hem emale. Among
he pa icipan s, 62.47% claimed o be amilia wi h
compu e g aphics, 25.57% had no p e ious expe ience
and 9.96% decla ed hemsel es p o essionals. We also
sampled da a ega ding he de ices used du ing he
expe imen s: 94.10% used a moni o , 4.30% used a
able , and 1.60% used a mobile phone. In addi ion,
he mos common sc een size was 1366 ×728 pixels
(42.01% o pa icipan s), minimum sc een size was 640
×360 pixels ( wo people), and a maximum o 2560 ×
1414 pixels (one pe son). Use s we e no awa e o he
pu pose o he beha io al expe imen .
P ocedu e
Subjec s a e shown a e e ence sample and a g oup
o candida e ma e ial samples. Each expe imen , HIT
in MTu k e minology, consis s o 23 unique e e ence
ma e ial samples o ials, 36 o which a e sen inels used
o de ec malicious o lazy use s. Use s a e asked o
“selec i e ma e ial samples which you belie e a e close
o he one shown in he e e ence image.” Addi ionally,
we ins uc hem o make hei selec ion in dec easing
o de o con idence. We le he use s pick i e candida e
ma e ials because jus one answe would p o ide spa se
esul s. We launched 25 HITs o each expe imen
and each HIT was answe ed by six di e en use s.
This esul ed in a o al o 27.000 nonsen inel ials,
12.000 belonging o he ou expe imen s analyzed in
he Resul s, and 15.000 o hem belonging o he i e
addi ional expe imen s discussed in he Discussion
(a o al o nine di e en expe imen s wi h 25 HITs each,
each HIT answe ed by six use s and 20 nonsen inel
ials pe HIT). Use s we e no allowed o epea he
same HIT.
The se o ma e ials in he candida e samples does
no a y ac oss HITs; howe e , he posi ion o each
sample is andomized o each ial. This has a wo- old
pu pose: i p e en s he use om memo izing he
posi ion o he samples, and i p e en s hem om
selec ing only he candida e samples ha appea a he
op o hei sc een. The e e ence samples do no epea
ma e ials du ing a HIT and he e e ence ma e ial is
always p esen among he candida e samples. Du ing
he expe imen , s imuli keep a cons an display size o
300 ×300 pixels o he e e ence, and o 120 ×120
pixels o he candida e s imuli (excep o some o he
addi ional expe imen s explained in Discussion whe e
bo h e e ence and candida e s imuli a e displayed a
ei he 300 ×300 pixels o 120 ×120 pixels). Figu e 3
shows a sc eensho wi h he g aphical use in e ace
du ing he beha io al expe imen s. On he le -hand
side, we can obse e he selec ion panel wi h he cu en
ial and he selec ion o he cu en ma e ials. The
igh -hand side displays he se o candida e ma e ials
whe eo use s can pick hei selec ion. Use s we e no
able o go back and edo an al eady answe ed ial, bu
hey could edi hei cu en selec ion o i e ma e ials
un il hey we e sa is ied wi h hei choice. Addi ionally,
once he 23 ials o he HIT a e answe ed, o ha e an
in ui ion abou he main ea u es ha humans use o
ma e ial ecogni ion, we asked he use : “Which isual
cues did you conside o pe o m he es ?”
To minimize wo ke un eliabili y, he use pe o ms a
b ie aining be o e he eal es (Welinde e al., 2010).
To a oid gi ing he use u he in o ma ion abou he
es , we use a di e en geome y (Ha an-3 Ha an
e al., 2016) du ing he aining phase. In his phase,
he i ems o he in e ace a e explained and he use is
gi en guidance on how o pe o m he es using jus a
ew images (Ga ces e al., 2014;Lagunas e al., 2018;
Rubins ein e al., 2010).
Sen inels
Each sen inel shows a andomly selec ed image om
he pool o candida es as he e e ence sample. We
conside use answe s o he sen inel as alid i hey pick
Downloaded om jo .a ojou nals.o g on 03/15/2021

Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 6
he igh ma e ial wi hin hei i e selec ions, ega dless
o he o de . We ejec ed use s who did no co ec ly
answe a leas one o he h ee sen inel ques ions. To
ensu e ha use s’ answe s we e well hough and ha
hey we e paying a en ion o he expe imen , we also
ejec ed use s ha ook less han 5 seconds pe ial (on
a e age). In he end, we adop a conse a i e app oach
and ejec ed 19.8% o he pa icipan s, ga he ing 21.660
answe s (9.560 belonging o he beha io al expe imen s
explained in he Resul s and 12.100 belonging o he
addi ional expe imen s explained in he Discussion).
Resul s
We in es iga e which ac o s ha e a signi ican
in luence on use pe o mance and on he ime hey
ook o comple e each ial in he ou expe imen s: Tes
HH, Tes HS, Tes SH, and Tes SS. The ac o s we
include a e: he e e ence geome y G e , he candida e
geome y Gcand, and he illumina ion o he e e ence
sample I e , as well as hei i s -o de in e ac ions
( ecall ha he illumina ion o he candida e samples
emains cons an in hese beha io al expe imen s). We
also include he O de o appea ance o each ial.
We use a gene al linea mixed model wi h a binomial
dis ibu ion o he pe o mance since i is well-sui ed
o bina y dependen a iables like ou s, and a nega i e
binomial dis ibu ion o he ime, which p o ides
mo e accu a e models han he Poisson dis ibu ion
by allowing he mean and a iance o be di e en .
Because we canno assume ha ou obse a ions a e
independen , we model he po en ial e ec o each
pa icula subjec iewing he s imuli as a andom
e ec . Because we ha e ca ego ical a iables among
ou p edic o s, we e-code hem o dummy a iables
o he eg ession. In all ou es s, we ix a signi icance
alue (P- alue) o 0.05. Finally, o ac o s ha p esen
a signi ican in luence, we u he pe o m pai wise
compa isons o all hei le els (leas signi ican
di e ence pai wise mul iple compa ison es ).
Analysis o use pe o mance and ime
In ou online beha io al expe imen s, we ely on he
op i e accu acy o measu e use pe o mance. This
me ic conside s an answe as co ec i he e e ence is
among he i e candida e ma e ials ha he use picked
in he ial. Because pa icipan s picked i e ma e ials
anked in descending o de o con idence, he op one
accu acy could also be conside ed o ou analysis.
Howe e , he ask hey ha e o sol e is no easy and
use s ha e an o e all op one accu acy o 9.21% which
yields spa se esul s. A andom selec ion would yield a
op one accu acy o 1% and a op i e accu acy o 5%.
Figu e 7. Le : Top fi e accu acy o each o he ou beha io al
expe imen . Cen e : Top fi e accu acy o each e e ence
geome y G e .Righ : Top fi e accu acy o he candida e
geome y Gcand. We can see how use s seem o pe o m
be e when he candida e and e e ence a e a high- equency
geome y. All plo s ha e a 95% confidence in e al. The names
ma ked wi h ∗a e ound o ha e s a is ically significan
diffe ences.
Influence o he geome y
The e is a clea e ec in use pe o mance when he
he geome y changes, ega dless i ha change happens
in he candida e (Gcand,P=0.005) o he e e ence
geome y (G e ,P<0.001). This inding is expec ed,
because he geome y plays a key ole in how a su ace
e lec s he incoming ligh and, he e o e, will ha e an
impac on he inal appea ance o he ma e ial. Figu e 7
shows use pe o mance in e ms o op i e accu acy
wi h a 95% con idence in e al when he e e ence and
candida e geome y change join ly (le ) o indi idually
(cen e and igh ). Use s seem o pe o m be e when
hey ha e o ecognize he ma e ial in a high- equency
geome y compa ed wi h a low- equency one. Those
esul s also sugges ha changes in he equencies o
he e e ence geome y may ha e a bigge impac on
use pe o mance han changes in he equencies o
he candida e geome y (i.e., use s pe o m be e wi h
a high- equency e e ence geome y and low- equency
candida e geome y, compa ed o a low- equency
e e ence geome y and a high- equency candida e
geome y).
Influence o he e e ence illumina ion
We obse e ha he illumina ion o he e e ence
image has a signi ican e ec on use pe o mance
(I e ,P<0.001). This inding is expec ed because all
he ma e ials in a scene a e e lec ing he ligh ha
eaches hem; he e o e, he changes in illumina ion
can signi ican ly in luence he inal appea ance o a
ma e ial, and how we pe cei e i (Bousseau e al., 2011).
Figu e 8, le , shows he op i e accu acy o each
e e ence illumina ion and g oups o illumina ions wi h
s a is ically indis inguishable pe o mance. We can
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 7
Figu e 8. Le : Top fi e accu acy o each e e ence illumina ion (I e ). We can see how use s seem o pe o m be e wi h
high- equency illumina ions (Uffizi,G ace,Ennis), while hei pe o mance is wo se wi h a low- equency illumina ion (Glacie ).
Addi ionally, hey ha e an in e media e pe o mance o medium- equency illumina ions (Doge and Pisa). Cen e : Top fi e accu acy
o each e e ence illumina ion when he candida e geome y (Gcand) changes. We can obse e how use s seem o pe o m
significan ly be e wi h a high- equency geome y (Ha an) and illumina ion. On he o he hand, o low- equency illumina ions,
changes in he candida e geome y yield s a is ically indis inguishable pe o mance. Righ : Top fi e accu acy o each e e ence
illumina ion when he e e ence geome y (G e ) changes. We can obse e how use s seem o pe o m significan ly be e o all
high- equency illumina ions, excep o G ace. The ho izon al lines unde he x-axis ep esen g oups o s a is ically indis inguishable
pe o mance. We can obse e how he g oups usually clus e high-, medium- and low- equency illumina ions. The e e ence
illumina ions ma ked wi h ∗deno e significan diffe ences in use pe o mance be ween geome ies o ha illumina ion. The e o
ba s co espond o a 95% confidence in e al.
obse e how use s seem o ha e be e pe o mance
when he su ace hey a e e alua ing has been li wi h a
high- equency illumina ion (Ennis,G ace,andU izi),
whe eas use s seem o pe o m wo se in scenes wi h a
low- equency illumina ion (Glacie ); use s show an
in e media e pe o mance wi h a medium- equency
illumina ion (Doge and Pisa). Mo eo e , we pe o med
a leas signi ican di e ence pai wise mul iple
compa ison es o ob ain g oups o illumina ions
wi h s a is ically indis inguishable pe o mance. These
g oups can be obse ed in Figu e 8, unde he x-axis. I
we ocus on I e we can see how high- (g een), medium-
(blue), and low- equency ( ed) illumina ions yield
g oups o simila pe o mance. The e is an addi ional
g oup o s a is ically indis inguishable pe o mance
ep esen ed in pink.
Influence o ial o de
The o de o appea ance o he ials du ing he
expe imen does no ha e a signi ican in luence in use s
pe o mance (O de ,P=0.391).
Fi s o de in e ac ions
We ind ha he in e ac ion be ween he candida e
geome y and he e e ence illumina ion has a
signi ican e ec on use pe o mance (Gcand ∗I e ,
P<0.001). Use s seem o pe o m be e wi h a
high- equency geome y (compa ed wi h a low-
equency one) when he e e ence s imuli ea u es a
high- equency illumina ion (I e =U izi,P=0.019;
I e =[G ace, Ennis], P<0.001). On he o he
hand, he e seems o be no signi ican changes in
pe o mance be ween a high- and low- equency
candida e geome y when he e e ence s imuli has
a medium- o low- equency illumina ion (I e =
Doge,P=0.453; I e =Pisa,P=0.381; I e =
Glacie ,P=0.770). We a gue ha use pe o mance
is d i en by he e e ence sample. When he e e ence
ma e ial is li wi h a low- equency illumina ion,
use s seem o no be able o p ope ly ecognize i .
The e o e, changes in he candida e geome y a e
no ele an o use pe o mance. These esul s can
be seen in Figu e 8, cen e . Fu he mo e, unde he
x-axis, we can obse e he g oups wi h s a is ically
indis inguishable pe o mance whe e high-, medium-,
and low- equency illumina ions yield g oups o simila
pe o mance.
We also ound ou ha he in e ac ion be ween he
e e ence geome y and he e e ence illumina ion has
a signi ican impac in use pe o mance (G e ∗I e ,
P=0.012). Use s seem o show be e pe o mance
o all illumina ions wi h a high- equency e e ence
geome y (G e =Ha an,I e =U izi,P=0.002; I e
=[Ennis, Pisa, Doge, Glacie ], P<0.001), excep o
G ace illumina ion (P=0.176), whe e he di e ences in
humans pe o mance a e s a is ically indis inguishable.
These esul s, oge he wi h he g oups o s a is ically
indis inguishable pe o mance, can be seen in Figu e 8,
igh .
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 8
Figu e 9. Visualiza ions o use answe s o each o he ou online beha io al expe imen s (namely, TEST HH, TEST HS, TEST SH, and TEST
SS) using he -STE algo i hm (Van De Maa en & Weinbe ge , 2012). The inse shows he colo o each ma e ial based on he colo
classifica ion p oposed by Lagunas e al. (2019). We can see how, o all expe imen s, ma e ials wi h simila colo p ope ies a e
g ouped oge he . Fu he mo e, i we explo e he colo clus e s indi idually, we can see how he e is a second-le el a angemen by
eflec ance p ope ies. These obse a ions sugges ha use s may be pe o ming a wo-s ep p ocess while ecognizing ma e ials
whe e fi s , hey so hem ou by colo , and second, by eflec ance p ope ies.
In gene al, we canno conclude ha he e a e
signi ican changes in pe o mance due o he
in e ac ion be ween he candida e and e e ence
geome y (G e ∗Gcand,P=0.407). Ne e heless, wi h
a low- equency e e ence geome y (G e =sphe e),
use s seem o pe o m signi ican ly be e wi h a
high- equency candida e geome y (Gcand =Ha an,
P=0.009).
Analysis o he ime spen on each ial
To accoun o ime, we measu e he numbe o
milliseconds ha passed since he ial loaded in hei
sc een and un il hey picked all i e ma e ials and
p essed he “Con inue” bu on.
Influence o ial o de
We ind ha he o de o he ials has a signi ican
in luence on he a e age ime use s spend o answe
hem (P<0.001). Use s spend mo e ime in he i s
ques ions and ha a e ew ials he a e age ime
hey spend becomes s able a a ound 20 seconds
pe ial ( ecall ha he o de does no in luence
pe o mance). This phenomenon is expec ed as
use s ha e o amilia ize wi h he expe imen du ing
he i s i e a ions. As he es ad ances, hey lea n
how o in e ac wi h i and he ime hey spend
becomes s able. Addi ional igu es and esul s on he
ac o s ha in luence he spen ime can be ound in
Appendix A.
High-le el ac o s d i ing ma e ial ecogni ion
In addi ion o he analysis, we also y o gain
in ui ion on which high-le el ac o s d i e ma e ial
ecogni ion, in es iga e how simple image s a is ics
and image his og ams co ela e wi h human answe s,
and analyze highly nonlinea s a is ics in ma e ial
classi ica ion asks by aining a deep neu al ne wo k.
Visualizing use answe s
To gain in ui ion on which high-le el ac o s
humans migh use while ecognizing ma e ials, we
use a s ochas ic iple embedding me hod called he
( -S uden s ochas ic iple embedding ( -STE) (Van
De Maa en & Weinbe ge , 2012) di ec ly on use
answe s. This me hod maps use answe s om hei
o iginal non-nume ical domain in o a wo-dimensional
space ha can be easily isualized ( ind addi ional
de ails in he Appendix B). Figu e 9 shows he
wo-dimensional embeddings a e applying he -STE
algo i hm o he answe s o each online beha io al
expe imen . Each poin in he embedding ep esen s 1
o he 100 ma e ials om he Lagunas e al. da ase .
The inse s show he colo o each ma e ial based on
he colo classi ica ion p oposed by Lagunas e al. We
can obse e how ma e ials a e clus e ed by colo and,
i we ocus in a single colo , hey seem o be clus e ed
by e lec ance p ope ies (e.g., in Tes HH, ed colo
clus e , we can obse e how on he le he e a e specula
ma e ials while on he igh he e a e di use ma e ials).
This inding sugges s ha use s ha e ollowed a
wo-s ep s a egy o ecognize he ma e ials, and ha
he high-le el ac o s d i ing ma e ial ecogni ion migh
be colo i s , and he e lec ance p ope ies second. A
he end o he HIT, use s we e asked o w i e he main
isual ea u es hey used o ecognize ma e ials. Ou o
368 unique use s om he expe imen s analyzed in he
Resul s, 273 answe ed ha hey ha e used he colo s,
and 221 answe ed ha hey elied on he e lec ions.
Among hem, 157 answe ed bo h colo and e lec ions
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 9
as some o he isual cues hey ha e used o pe o m
he ask. This obse a ion, oge he wi h he -STE
isualiza ion, s eng hens he hypo hesis o a wo-s ep
s a egy.
Image s a is ics
P e ious s udies ocused on simple image s a is ics
as an a emp o u he unde s and ou isual sys em
(Adelson, 2008;Mo oyoshi e al., 2007). Ne e heless,
i is a gued whe he ou isual sys em ac ually
de i es any aspec s o ma e ial pe cep ion using such
simple s a is ics (Ande son & Kim, 2009;Kim &
Ande son, 2010;Olkkonen & B aina d, 2010). We
es ed he co ela ion be ween he i s ou s a is ical
momen s o he luminance (conside ed as he a io:
L=0.3086 ∗R+0.6094 ∗G+0.0820 ∗B), he pixel
in ensi y o each colo channel independen ly, and he
join RGB pixel in ensi y, di ec ly agains use s op i e
accu acy. To measu e co ela ion we employ a Pea son
Pand Spea man Sco ela ion es . We ound ou ha
he e is li le o no co ela ion, excep o he s anda d
de ia ion o he join RGB pixel in ensi y whe e
P2=0.43 (P<0.001) and S2=0.50 (P<0.001).
Addi ional in o ma ion can be ound in Appendix C.
Image his og ams
We also compu e he his og ams o he RGB pixel
in ensi y, o he luminance, o a Gaussian py amid (Lee
& Lee, 2016), o a Laplacian py amid (Bu & Adelson,
1983), and o log-Gabo il e s designed o simula e
he ecep i e ield o he simple cells o he P ima y
Visual Co ex (V1) (Fische e al., 2007). To see how
such his og ams would pe o m a classi ying ma e ials,
we ain a suppo ec o machine (SVM) ha akes he
image his og am as he inpu and classi ies he ma e ial
in ha image. We use a adial basis unc ion ke nel
(o Gaussian ke nel) in he SVM. We use all image
his og ams ha do no ea u e Ha an geome y as he
aining se and lea e he ones wi h Ha an as es se .
In he end, he bes pe o ming SVM uses he RGB
image his og am as he inpu and achie es a 24.17% op
i e accu acy in he es se .
In addi ion, we compa e he p edic ions o each
SVM di ec ly agains human answe s. Fo each
e e ence s imuli we compa e he i e selec ions o
he use agains he i e mos likely SVM ma e ial
p edic ions o ha s imuli. The bes SVM uses he
his og ams o V1-like subband il e s and ag ees wi h
humans 6.36% o he ime. Mo eo e , we compa e
his og am simila i ies agains human answe s using a
Χ2his og am dis ance (Pele & We man, 2010). Fo a
e e ence image s imuli we measu e i s simila i y agains
all possible candida e image s imuli and compa e he
closes i e agains pa icipan s answe s. The Gaussian
py amid his og am ob ained he bes esul , ag eeing
wi h humans 6.29% o he ime. These esul s show how
simple s a is ics, and highe -o de image his og ams
seem no o be capable o ully cap u ing human
beha io . We ha e added addi ional esul s on he
SVMs and human ag eemen in Appendix C.
Image equencies
To unde s and i humans’ pe o mance could be
explained by aking in o accoun he spa ial equency
o he e e ence s imuli, a hei iewed size, we ha e
added he HFC measu e, and he i s ou s a is ical
momen s o he e e ence s imuli magni ude spec um
o he ac o s analyzed in he Resul s. We ound ha he
Skewness (P<0.001) and Ku osis (P<0.001) o he
magni ude spec um seem o ha e a signi ican in luence
on humans pe o mance; howe e , hey p esen a e y
small e ec size.
Highly nonlinea models
Recen s udies sugges ha , o unde s and wha
su ounds us, ou isual sys em is doing an e icien
nonlinea encoding o he p oximal s imulus ( he
image inpu o ou isual sys em) and ha highly
nonlinea models migh be able o be e cap u e human
pe cep ion (Delanoy e al., 2020;Fleming&S o s,
2019). Inspi ed by his hypo hesis, we ha e ained a
deep neu al ne wo k called ResNe (He e al., 2016)
using a loss unc ion sui able o classi y he ma e ials
in he Lagunas e al. da ase . The images ea u e he
same illumina ions as he e e ence s imuli. We le
ou he images ende ed wi h Ha an geome ies o
alida ion and es ing pu poses, and use he es du ing
aining. To know which ma e ial he ne wo k classi ies
we add a so max laye a he end o he ne wo k. The
so max laye ou pu s he p obabili y o he inpu
image o belong o each ma e ial in he da ase . In
compa ison, he model used by Lagunas e al. does
no ha e he las ully connec ed and so max laye ,
and i is ained using a iple loss unc ion aiming o
simila i y ins ead o classi ica ion. A he end o he
aining, he model achie es a op-5 accu acy o 89.63%
on he es se , sugges ing ha such models a e ac ually
capable o ex ac ing meaning ul ea u es om labeled
p oximal image da a (addi ional de ails on he aining
can be ound in Appendix D). To gain in ui ion on how
he ne wo k has lea ned, we ha e used he Uni o m
Mani old App oxima ion and P ojec ion algo i hm
(McInnes & Healy, 2018). This algo i hm aims o
dec ease he dimensionali y o a se o ea u e ec o s
while main aining he global and local s uc u e o hei
o iginal mani old. Figu e 10 shows a wo-dimensional
isualiza ion o he es se ob ained using he 128
ea u es o he ully connec ed laye be o e so max. We
can obse e how ma e ials seem o be g ouped i s by
colo and hen by i s e lec ance p ope ies sugges ing
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 16
Lagunas, M., Ga ces, E., & Gu ie ez, D. (2018).
Lea ning icons appea ance simila i y. Mul imedia
Tools and Applica ions, 1–19.
Lagunas, M., Malpica, S., Se ano, A., Ga ces, E.,
Gu ie ez, D., & Masia, B. (2019). A Simila i y
Measu e o Ma e ial Appea ance. ACM
T ansac ions on G aphics (P oc. SIGGRAPH, 38(4).
Lee, S., & Lee, D. (2016). Fusion o IR and Visual
Images Based on Gaussian and Laplacian
Decomposi ion Using His og am Dis ibu ions
and Edge Selec ion. Ma hema ical P oblems in
Enginee ing, 2016.
Leloup, F. B., Poin e , M. R., Du é, P., & Hanselae ,
P. (2010). Geome y o illumina ion, luminance
con as , and gloss pe cep ion. JOSA, A27(9),
2046–2054.
Li, W., & F i z, M. (2012). Recognizing ma e ials
om i ual examples. In Eu opean Con e ence on
Compu e Vision (ECCV). Sp inge , 345–358.
Maloney, L. T., & B aina d, D. H. (2010). Colo and
ma e ial pe cep ion: Achie emen s and challenges.
Jou nal o Vision (JOV), 10(9), 19–19.
Mao,R.,Lagunas,M.,Masia,B.,&Gu ie ez,D.
(2019). The e ec o mo ion on he pe cep ion
o ma e ial appea ance. P oceedings o he ACM
symposium on applied pe cep ion (SAP). (p. 9).
ACM.
Ma low, P. J., Kim, J., & Ande son, B. L. (2012). The
pe cep ion and mispe cep ion o specula su ace
e lec ance. Cu en Biology, 22(20), 1909–1913.
Ma usik, W., P is e , H., B and, M., & McMillan, L.
(2003). A Da a-D i en Re lec ance Model. ACM
T ansac ions on G aphics (TOG), 22(3), 759–769.
McInnes, L., & Healy, J. (2018). Umap: Uni o m
mani old app oxima ion and p ojec ion
o dimension educ ion. a Xi p ep in
a Xi :1802.03426.
Mo oyoshi, I., Nishida, S., Sha an, L., & Adelson, E.
H. (2007). Image s a is ics and he pe cep ion o
su ace quali ies. Na u e, 447(7141), 206–209.
Nagai, T., Ma sushima, T., Koida, K., Tani, Y.,
Ki azaki, M., & Nakauchi, S. (2015). Tempo al
p ope ies o ma e ial ca ego iza ion and ma e ial
a ing: isual s non- isual ma e ial ea u es. Vision
Resea ch, 115, 259270. Pe cep ion o Ma e ial
P ope ies (Pa II).
Nishida, S., & Shinya, M. (1998). Use o image-based
in o ma ion in judgmen s o su ace- e lec ance
p ope ies. JOSA A15,12, 2951–2965.
Obein, G., Knoblauch, K., & Viéo , F. (2004).
Di e ence scaling o gloss: Nonlinea i y,
binocula i y, and cons ancy. Jou nal o Vision
(JOV), 4(9), 4–4.
Oli a, A., & To alba, A. (2001). Modeling he shape
o he scene: A holis ic ep esen a ion o he spa ial
en elope. In e na ional Jou nal o Compu e Vision
(IJCV), 42(3), 145–175.
Olkkonen, M., & B aina d, D. H. (2010). Pe cei ed
glossiness and ligh ness unde eal-wo ld
illumina ion. Jou nal o Vision (JOV), 10(9),
5–5.
Olkkonen, M., & Da id, H. B., (2011). Join e ec s
o illumina ion geome y and objec shape in he
pe cep ion o su ace e lec ance. i-Pe cep ion2,9,
1014–1034.
Palme , S. (1975). Visual pe cep ion and wo ld
knowledge: No es on a model o senso y-cogni i e
in e ac ion. Explo a ions in Cogni ion, 279–307.
Pele, O., & We man, M. (2010). The quad a ic-chi
his og am dis ance amily. In Eu opean con e ence
on compu e ision. Sp inge , 749–762.
Pizlo, Z. (2001). Pe cep ion iewed as an in e se
p oblem. Vision Resea ch41,24, 3145–3161.
Ramamoo hi, R., & Han ahan, P. (2001). An e icien
ep esen a ion o i adiance en i onmen maps. In
P oceedings o he Annual con e ence on Compu e
G aphics and In e ac i e Techniques. 497–500.
Rubins ein, M., Gu ie ez, D., So kine, O., & Shami , A.
(2010). A Compa a i e S udy o Image Re a ge ing.
ACM T ansac ions on G aphics (P oc. SIGGRAPH
Asia 2010), 29(6), 160:1–160:10.
Scha ali zky, F., & Zisse man, A. (2001). Viewpoin
in a ian ex u e ma ching and wide baseline
s e eo. In P oceedings o he IEEE In e na ional
Con e ence on Compu e Vision (ICCV),Vol. 2.
IEEE, 636–643.
Schille , P. H., Finlay, B. L., & Volman, S. F. (1976).
Quan i a i e s udies o single-cell p ope ies
in monkey s ia e co ex. I. Spa io empo al
o ganiza ion o ecep i e ields. Jou nal o
Neu ophysiology, 39(6), 1288–1319.
Schlü e , N., & Faul, F. (2019). Visual shape pe cep ion
in he case o anspa en objec s. Jou nal o Vision
(JOV), 19(4), 24–24.
Se ano, A., Gu ie ez, D., Myszkowski, K., Seidel, H.-
P., & Masia, B. (2016). An In ui i e Con ol Space
o Ma e ial Appea ance. ACM T ansac ions on
G aphics (TOG), 35(6), A icle 186 ( No .2016),
186:1–186:12 pages.
Sè e, R. (1993). P oblems connec ed wi h he concep
o gloss. Colo Resea ch & Applica ion, 18(4),
241–252.
Sha an, L., Rosenhol z, R., & Adelson, E. (2009).
Ma e ial pe cep ion: Wha can you see in a b ie
glance? Jou nal o Vision (JOV), 9(8), 784–
784.
Downloaded om jo .a ojou nals.o g on 03/15/2021

Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 17
Sha an, L., Rosenhol z, R., & Adelson, E. H. (2008).
Eye mo emen s o shape and ma e ial pe cep ion.
Jou nal o Vision (JOV), 8(6), 219–219.
Sun, T., Se ano, A., Gu ie ez, D., & Masia, B. (2017).
A ibu e-p ese ing gamu mapping o measu ed
BRDFs. In Compu e G aphics Fo um,Vol. 36.
Wiley Online Lib a y, 47–54.
Szegedy, C., Vanhoucke, V., Io e, S., Shlens, J.,
& Wojna, Z. (2015). Re hinking he incep ion
a chi ec u e o compu e ision. a Xi .
Thompson, W., Fleming, R., C eem-Regeh , S., &
S e anucci, J. K. (2011). Visual Pe cep ion om
a Compu e G aphics Pe spec i e (1s ed.).A.K.
Pe e s, L d., Na ick, MA, USA.
Thompson, W., Fleming, R., C eem-Regeh , S., &
S e anucci, J. K. (2016). Visual pe cep ion om a
compu e g aphics pe spec i e. AK Pe e s/CRC
P ess.
Van De Maa en, L., & Weinbe ge , K. (2012).
S ochas ic iple embedding. In Machine Lea ning
o Signal P ocessing (MLSP), 2012 IEEE
In e na ional Wo kshop on. IEEE, 16.
Vango p, P., Lau ijssen, J., & Du é, P. (2007). The
In luence o Shape on he Pe cep ion o Ma e ial
Re lec ance. ACM T ansac ions on G aphics
(TOG), 26(3), A icle 77 (July2007).
Ví a, R., & Filip, J. (2016). Minimal sampling o
e ec i e acquisi ion o aniso opic BRDFs. In
Compu e G aphics Fo um,Vol. 35. Wiley Online
Lib a y, 299–309.
Wang, T.-C., Zhu, J.-Y., Hi oaki, E., Chand ake ,
M., E os, A. A., & Ramamoo hi, R. (2016). A
4D ligh - ield da ase and CNN a chi ec u es o
ma e ial ecogni ion. In Eu opean Con e ence on
Compu e Vision. Sp inge , 121–138.
Welinde , P., B anson, S., Pe ona, P., & Belongie, S. J.
(2010). The mul idimensional wisdom o c owds. In
Ad ances in Neu al In o ma ion P ocessing Sys ems
(Neu IPS). 2424–2432.
Zhang, F., Ridde , H., & Pon , S. (2015). The in luence
o ligh ing on isual pe cep ion o ma e ial
quali ies. In Human Vision and Elec onic Imaging
XX, Vol. 9394. In e na ional Socie y o Op ics and
Pho onics, 93940Q.
Appendix A: Addi ional esul s on
he influence o ime
Addi ional de ails on he ime ha each pa icipan
spen doing he online beha io al expe imen . In
Figu e 16 we can see how he ime spen o answe
each ial becomes s able as he beha io al expe imen
ad ances.
In luence o e e ence illumina ion: The e e ence
illumina ion I e in luences he ime use s spend o
answe each ial (P=0.001). Use s spend mo e ime
when he s imuli a e li wi h Ennis illumina ion while
hey a e he as es when he illumina ion is Doge.
We did no ind a signi ican in luence o he
e e ence geome y G e o candida e geome y Gcand
in he a e age ime each use spen o answe each ial.
Fi s o de in e ac ions: We obse e ha use s ake
signi ican ly longe o answe he ials when bo h he
e e ence geome y and he candida e geome y change
(G e ∗Gcand,P=0.001). This happens in he case
whe e he e e ence geome y has mos ly low spa ial
equency con en and he candida e geome y changes
(Gcand =sphe e,P=0.002); and when he e e ence
has mos ly low spa ial equency (G e =sphe e,P=
0.001) and he candida e geome y changes.
Appendix B: Addi ional de ails on
he -STE algo i hm
The -STE algo i hm aims o ob ain an n-
dimensional embedding ha sa is ies as many
quali a i e compa isons o he ype “A is mo e simila
o B han C” as possible. In ou case, a wo-dimensional
embedding which is easie o isualize. Ne e heless, in
he use s udies, we ha e asked pa icipan s o selec i e
ma e ials om a pool o candida es and we do no ha e
such quali a i e compa isons. Howe e , we can assume
ha he selec ion o he use s will be close (mo e
simila ) o he e e ence han any o he ma e ial ha
was no selec ed. Based on his assump ion, we gene a e
iple s whe e he use selec ion is mo e simila o he
e e ence ma e ial han any o he andom ma e ial ha
is no wi hin he 5 selec ed ma e ials. We epea his
p ocess en imes o each o he 5 ma e ials selec ed by
Figu e 16. A e age ime he use s spen o each ial acco ding
o he o de o appea ance du ing he online beha io al
expe imen . We can obse e how, as he use p og esses
h ough he expe imen , he ime spen on each ial becomes
s able. The e o ba s co espond o a 95% confidence in e al.
Downloaded om jo .a ojou nals.o g on 03/15/2021
Jou nal o Vision (2021) 21(2):2, 1–18 Lagunas, Se ano, Gu ie ez, & Masia 18
he use making su e ha he new andomly sampled
ma e ial has no been andomly selec ed al eady no
ha i belongs o he pool o 5 selec ed ma e ials.
To un he -STE we se a lea ning a e o 1, and an
α=25 (deg ees o eedom o he S uden - ke nel).
Addi ionally, we apply a loga i hmic ans o ma ion o
he loss alue o he -STE. Those pa ame e s a e he
same o he answe s o he ou expe imen s.
Appendix C: Addi ional de ails on
image s a is ics
To measu e he co ela ion be ween image s a is ics
and use s pe o mance we employ a Pea son Pand
Spea man Sco ela ion es wi h a signi icance alue
(P- alue) o 0.05. The alue Pn ep esen s he Pea son
co ela ion o he n h s a is ical momen (same applies
o he Spea man Snco ela ion).
Luminance: We analyze i he momen s o he
luminance o each ma e ial image ha e a di ec
in luence on use s pe o mance. We ound ha he
momen s o he luminance a e no co ela ed wi h use s
pe o mance: P1=−0.14 (P=0.17),S1=−0.15
(P=0.15),P2=0.02 (P=0.83),S2=−0.03 (P=
0.78),P3=0.03 (P=0.77),S3=0.03 (P=0.78),
P4=0.01 (P=0.94),S4=0.05 (P=0.65).
RGB image: We analyze i he momen s o he
join RGB in ensi y o each ma e ial image ha e
a di ec in luence on use s pe o mance. We ound
ha he momen s o he join RGB in ensi y ha e
li le o no co ela ion wi h use s pe o mance
excep o he s anda d de ia ion: P1=−0.02
(P=0.79),S1=−0.06 (P=0.51),P2=0.43
(P<0.001),S2=0.50 (P<0.001),P3=0.16
(P=0.09),S3=0.22 (P=0.02),P4=−0.1
(P=0.30),S4=−0.06 (P=0.52).
We also es ed ou he co ela ion o each channel
and ound ou ha o all he channels he e is no
co ela ion o any o he i s 4 s a is ical momen s.
Red channel: On he ed channel he e seems o
be a sligh posi i e linea co ela ion be ween he
ou h momen (ku osis) and use s pe o mance. All
he o he s a is ics show no signi ican co ela ion:
P1=−0.10 (P=0.29), S1=−0.08 (P=0.42),
P2=0.03 (P=0.60), S2=−0.02 (P=0.87),
P3=0.07 (P=0.46), S3=0.07 (P=0.51), P4=0.20
(P=0.04), S4=0.15 (P=0.13).
G een channel: The e is no co ela ion be ween any
s a is ics on he g een channel: P1=−0.04 (P=0.66),
S1=−0.0.(P=0.74), P2=0.03 (P=0.55), S2=0.04
(P=0.67), P3=0.05 (P=0.64), S3=0.06 (P=0.53),
P4=0.05 (P=0.63), S4=0.01 (P=0.94).
Blue channel: Simila o he g een channel, he
blue does no show any co ela ion o he i s 4
s a is ical momen s: P1=0.03 (P=0.72), S1=−0.004
(P=0.93), P2=0.06 (P=0.52), S2=0.01 (P=0.95),
P3=0.13 (P=0.19), S3=0.10 (P=0.30), P4=0.16
(P=0.11), S4=−0.05 (P=0.61).
Addi ional esul s on he SVMs and his og am
simila i y
We ha e ained a o al o 6 SVM models, each
o hem using a di e en inpu : RGB pixel in ensi y,
luminance in ensi y, Gaussian py amid pixel in ensi y
(Lee & Lee, 2016), Laplacian py amid pixel in ensi y
(Bu & Adelson, 1983), joining he Gaussian
and Laplacian py amids, and using log-Gabo
il e s (Fische e al., 2007). Fo each o hem he
SVM achie ed a op-5 accu acy in he es se o :
24.17%, 15.16%, 22.50%, 6.33%, 7.52%, and 16.33%,
espec i ely. In addi ion, we ha e compa ed how he
SVM p edic ions ag eed wi h humans’ answe s om
he online beha io al expe imen s. Fo each SVM he
ag eemen is: 4.24%, 4.33%, 4.34%, 5.04%, 4.97%, and
6.36% espec i ely. Las , we ha e also compu ed he
his og am simila i y using a Χ2dis ance. Then, we
ha e aken he i e closes samples and compa ed ha
wi h human answe s. We do ha o each o he i e
di e en his og ams and each achie es an ag eemen
o : 5.95%, 5.45%, 6.29%, 4.97%, 5.04%, and 5.07%
espec i ely.
Appendix D: Addi ional de ails on
ResNe aining
To ain he 35 laye s ResNe (34 o he o iginal
model plus an addi ional ully connec ed) (He e al.,
2016) we ha e employed he da ase in oduced by
Lagunas e al. (Lagunas e al., 2019), which con ains
ende ings o ma e ials wi h di e en illumina ions and
geome ies. We keep he images ende ed wi h Ha an-3
geome y o alida ion pu poses and Ha an geome y
o es ing. All he o he images a e used o aining.
To ain he model o classi y ma e ials we use a so
c oss-en opy loss whe e samples ha do no belong
o he same class a e penalized (Szegedy e al., 2015).
The loss unc ion akes he p obabili ies ou pu o
he so max laye and penalizes when hey gi e a high
p obabili y o he ma e ials ha do no belong o he
inpu image. The images inpu o he model a e esized
o 224 ×224 pixels. The pa ame e s o he model a e
ini ialized using a p e ained e sion on ImageNe
da ase (Deng e al., 2009). We use he ADAM
algo i hm (Kingma & Ba, 2014) as he op imize . The
model has been ained du ing 50 i e a ions s a ing a
alea ning a eo 10
−3and decayed by a ac o o 10 a
he i e a ion 20, 35, and 45; he ba ch-size was se o
64 images. We use he PyTo ch amewo k and use an
N idia 2080Ti GPU.
Downloaded om jo .a ojou nals.o g on 03/15/2021