Expe Sys ems Wi h Applica ions 199 (2022) 116700
A ailable online 19 Ma ch 2022
0957-4174/© 2022 Else ie L d. All igh s ese ed.
Con en s lis s a ailable a ScienceDi ec
Expe Sys ems Wi h Applica ions
jou nal homepage: www.else ie .com/loca e/eswa
On alida ing web in o ma ion ex ac ion p oposals
Pa icia Jiménez∗, Ra ael Co chuelo
Uni e sidad de Se illa, ETSI In o má ica, A da. Reina Me cedes s/n, Se illa, E-41012, Spain
ARTICLE INFO
Keywo ds:
Web in o ma ion ex ac o s
Valida ion me hod
ABSTRACT
Many people who ha e o make in o med decisions in oday’s always-on cul u e use in o ma ion ex ac o s
o eed hei sys ems wi h in o ma ion ha comes om human- iendly documen s. Un o una ely, many
p oposals ha alida e in o ma ion ex ac o s ha e de iciencies ha make i di icul o pe o m homogeneous
compa isons, con i m o e u e pe o mance hypo heses, o d aw unbiased conclusions. Consequen ly, i is
e y di icul o selec he bes -pe o ming p oposal on a sound basis. The s a e-o - he-a alida ion me hod
o e comes many de iciencies in he p e ious p oposals, bu s ill o e looks he ollowing issues: comple eness
o he alida ion da ase s, ha is, whe he hey p o ide a comple e se o anno a ions o no ; s uc u e
o he in o ma ion, ha is, whe he hey check he s uc u e o he eco d ins ances ex ac ed o jus he
a ibu e ins ances; and, inally, how ex ac ions and anno a ions a e ma ched. The decisions made ega ding
he p e ious issues ha e an impac on he e ec i eness esul s. In his a icle, we ha e exhaus i ely analysed
he li e a u e and we ha e also highligh ed he main weaknesses o ackle. We p esen a guideline and a me hod
o compu e he e ec i eness, which complemen s and enhances he s a e-o - he-a alida ion me hod.
1. In oduc ion
Today’s always-on cul u e is pushing o wa d a new gene a ion
o sys ems ha help people make in o med decisions building on
he in o ma ion ha hey ex ac om human- iendly documen s on
he Web. In he li e a u e, he e a e many p oposals o implemen
in o ma ion ex ac o s (Baumga ne e al.,2018;Chang e al.,2006;
Fe a a e al.,2014;Sleiman & Co chuelo,2013;Tu mo e al.,2006).
Many o hem equi e he use o p o ide a lea ning da ase wi h some
sample documen s om which ex ac ion ules a e lea n ; depending on
whe he he lea ning da ase is equi ed o p o ide anno a ions o no ,
he lea ning me hod is said o be supe ised o unsupe ised, espec-
i ely. The e a e also heu is ic-based p oposals ha use buil -in ules
ha ha e p o en o wo k well wi h many di e en documen s. The
in o ma ion ex ac ed by a supe ised p oposal has use -de ined labels
wi h a meaning; con a ily, he in o ma ion ex ac ed by unsupe ised
o heu is ic-based p oposals ha e compu e -gene a ed labels ha mus
be mapped on o use -de ined labels la e .
Jiménez e al. (2016) ound ha he me hods used o alida e in o -
ma ion ex ac o s a e o en poo ly documen ed and ha e some common
de iciencies ha may ha e a signi ican impac on he expe imen al
esul s. This is pa icula ly impo an inso a i hampe s con i ming
o e u ing he esul s, makes he compa isons wi h o he p oposals
he e ogeneous, and may easily bias he conclusions. They de ised
ARIEX, which is he s a e-o - he-a me hod o alida e in o ma ion
∗Co esponding au ho .
E-mail add esses: [email p o ec ed] (P. Jiménez), [email p o ec ed] (R. Co chuelo).
ex ac ion p oposals. Un o una ely, we ha e ound ou ha he e a e
h ee impo an issues ha i does no ake in o accoun , namely: (a)
Comple eness o he alida ion da ase s, ha is, o wha ex en he
alida ion da ase has been anno a ed. They a e commonly ully an-
no a ed in he con ex o supe ised p oposals, bu pa ially anno a ed
in he con ex o unsupe ised and heu is ic-based p oposals. (b) The
s uc u e o he in o ma ion and how he alida ion p ocess ook i
in o accoun . The in o ma ion is commonly s uc u ed as collec ions o
(possibly nes ed) a ibu es o eco d ins ances. And (c) how ex ac ions
and anno a ions a e ma ched. The ma ching s a egies a e commonly
classi ied as exac , con ains, o o e lapping ma chings. The decisions
made ega ding he p e ious issues ha e an impac on he way ha
con usion ma ices a e compu ed, which, in u n, has an impac on he
e ec i eness measu es used o make compa isons and ankings.
In his a icle, we ad oca e ha esea che s who use ARIEX o
alida e hei p oposals mus also epo on he comple eness o hei
alida ion da ase s, on how hey ake he s uc u e o he in o ma ion
in o accoun , and how hey compu e ma ches amongs he anno a ions
and he ex ac ions; we also desc ibe a me hod o compu e con usion
ma ices ha akes he p e ious decisions in o accoun . This cons i-
u es a no el con ibu ion since i complemen s he s a e-o - he-a
alida ion me hod wi h addi ional guidelines ega ding issues ha we e
o e looked p e iously.
The es o he a icle is o ganised as ollows: in Sec ion 2, we epo
on he mos closely- ela ed p oposals and he ex en o which hey ha e
h ps://doi.o g/10.1016/j.eswa.2022.116700
Recei ed 17 No embe 2020; Recei ed in e ised o m 7 Feb ua y 2022; Accep ed 19 Feb ua y 2022
Expe Sys ems Wi h Applica ions 199 (2022) 116700
2
P. Jiménez and R. Co chuelo
o e looked he h ee issues ha we e men ioned p e iously; in Sec-
ion 3, we p esen some p elimina y concep s ha a e used h oughou
he a icle; in Sec ion 4, we desc ibe he issues and p opose a guideline
o ex end ARIEX; in Sec ion 5, we p esen ou me hod o compu e
he con usion ma ix and he e ec i eness measu es acco ding o he
guideline; in Sec ion 6, we epo on he esul s o ou expe imen a ion;
inally, we p esen ou conclusions in Sec ion 7.
2. Rela ed wo k
Valida ing an in o ma ion ex ac o amoun s o con on ing i wi h
a se ies o documen s and checking he ex en o which i can ex ac
in o ma ion om hem. The esul s allow o compa e an ex ac o o
i s compe i o s and o ank hem acco ding o di e en e ec i eness
measu es.
The i s eco ds o o mal me hods o alida e in o ma ion ex ac-
o s da e back o he end o he las cen u y (Chincho e al.,1993;
Hi schman,1998;Lehne & Sundheim,1991). They we e in ended o
semi-au oma ically compa e p oposals o ex ac in o ma ion om ee-
ex documen s in he con ex o he well-known MUC con e ences.
La e , La elli e al. (2004), I eson e al. (2005), and La elli e al.
(2008) poin ed ou a ew common mis akes in he p e ious me hods
and imp o ed on hem. Since hen, he e has been an inc easing in e -
es in ex ac ing da a om semi-s uc u ed documen s (Baumga ne
e al.,2018;Chang e al.,2006;Fe a a e al.,2014;Roldán e al.,
2020;Sleiman & Co chuelo,2013;Tu mo e al.,2006) in which he
in o ma ion is w i en in o ms, lis ings, o ables. We ha e e iewed
mos o he p oposals and ou conclusion is ha hey do no ypically
un eil some key de ails ega ding he alida ion p ocess. Mos o he
a icles simply p esen he esul s o a new app oach and he e is
a ely a de ailed analysis o ensu e ha he same me hodology is used
ac oss di e en expe imen s. Fu he mo e, he alida ion p ocess is
ypically poo ly documen ed and he e exis s much he e ogenei y in
he expe imen al se ings; in a ew cases, no expe imen al esul s a e
epo ed a all (Baumga ne e al.,2007;Raposo e al.,2002;Sahugue
& Aza an ,2001).
Jiménez e al. (2016) de ised ARIEX, which is he s a e-o - he-a
me hod o alida e in o ma ion ex ac o s. I allows o check hem
on a collec ion o well-known da ase s and allows o compa e he
e ec i eness esul s as homogeneously as possible and o ank hem as
au oma ically as possible. Howe e , ou ecen expe ience wi h de is-
ing new in o ma ion ex ac o s (Jiménez & Co chuelo,2016a,2016b;
Jiménez e al.,2021,2020;Roldán e al.,2017,2020,2021) e eals
ha i can be u he imp o ed o ake some addi ional issues in o
accoun , namely: (a) whe he he alida ion da ase s a e comple ely o
pa ially anno a ed; (b) whe he hey con ain eco d alues o no and
how hei s uc u e is aken in o accoun o compu e he e ec i eness
measu es; and (c) how he ma chings amongs he anno a ions and he
ex ac ions a e compu ed.
Rega ding he comple eness o he alida ion da ase s, we can make
a dis inc ion be ween p oposals ha seem o ha e been alida ed
manually (Pa k & Ba bosa,2007) o au oma ically (C escenzi e al.,
2001;F ei ag,2000;Hsu & Dung,1998;Jiménez & Co chuelo,2016a;
Kayed & Chang,2010;Kushme ick e al.,1997;Shen & Ka ge ,2007;
de Si e & Daelemans,2003;Sleiman & Co chuelo,2014;Zhang e al.,
2015). In he i s case, a use analyses he ex ac ions made by he
echnique, decides on whe he hey a e co ec o no , and hen com-
pu es some e ec i eness measu es; clea ly his me hod is subjec i e
and may in oduce biases e y easily. In he second case, a alida ion
da ase is p o ided and he ex ac ions a e somehow ma ched wi h he
anno a ions au oma ically so ha he e ec i eness measu es can also
be compu ed au oma ically; clea ly an au oma ic me hod is p e e able
and less biased. Un o una ely, some au ho s did no p o ide any clues
on how hey c ea ed hei alida ion da ase s (Ál a ez e al.,2010;
Hogue & Ka ge ,2005;I mak & Suel,2006;Zhai & Liu,2005). One
migh assume ha hey we e anno a ed comple ely o compu e he
e ec i eness measu es, bu he au ho s emphasised ha hey spen e y
li le ime on anno a ing and supe ising hei p oposals; hus, i is no
clea how he anno a ions we e made and whe he hey we e pa ial o
comple e. I mak and Suel (2006) jus men ioned ha hei echnique
wo ked well wi h a single andomly chosen documen o aining and
en documen s o alida ion. Hogue and Ka ge (2005) did no p o ide
any e ec i eness measu es; hey only e alua ed i hei p oposal was
able o lea n a good in o ma ion ex ac o o one si e. Al hough he
da ase s we e anno a ed, how hey pe o med he alida ion was no
documen ed. I is unclea i he e alua ion by Ál a ez e al. (2010)
was manual o no since hei echnique is unsupe ised and hey
epo ed on p ecision and ecall on a se o 200 si es ha we e no doc-
umen ed; nei he was i documen ed how he e ec i eness measu es
we e compu ed au oma ically. Zhai and Liu (2005) used 49 si es om
which 72 documen s we e collec ed. No clue was p o ided ega ding
whe he hey lea n and alida ed hei in o ma ion ex ac o s wi h
di e en documen s om he same si e o how he alida ion da ase
was c ea ed. The au ho s epo ed on co ec ness, which was su ely
compu ed au oma ically, bu i is no clea i he alida ion da ase was
comple e o no since hey emphasised ha hei goal was o educe he
anno a ion e o . Un o una ely, none o he p oposals su eyed docu-
men ed whe he spu ious in o ma ion was ex ac ed o no . Tha is, i
is likely ha he ex ac ions did no ma ch some anno a ions pe ec ly.
In such a case, he in o ma ion ha does no ma ch he anno a ions
may be spu ious o coun as alse posi i es. I he alida ion da ase
was pa ially anno a ed, we canno make su e i ha in o ma ion is
ac ually a alse posi i e o a missed ue posi i e, so ha coun ing i as
spu ious in o ma ion seems o make sense.
Rega ding he s uc u e o he in o ma ion, none o he p oposals
p o ide any de ails ega ding how hey alida ed he s uc u e o he
eco ds. Some p oposals a e supposed o lea n a empla e o he
documen s in a si e (C escenzi e al.,2001;Kayed & Chang,2010),
o he s lea n a empla e o eco ds and a ibu es (Sleiman & Co chuelo,
2014), o he can iden i y da a egions only (Sleiman & Co chuelo,
2013), o he s can ex ac eco ds only (Pa k & Ba bosa,2007;Shen &
Ka ge ,2007), o he s ex ac ela ions be ween a ibu es only (Zhang
e al.,2015), o he s can ex ac only a ibu es (F ei ag,2000;de Si -
e & Daelemans,2003), and many o hem can ex ac eco ds and
a ibu es wi hou an explici schema (Ál a ez e al.,2010;Hogue
& Ka ge ,2005;Hsu & Dung,1998;I mak & Suel,2006;Jiménez
& Co chuelo,2016a;Kushme ick e al.,1997;Zhai & Liu,2005).
Un o una ely, i is di icul o guess how he e ec i eness measu es
we e compu ed acco ding o he ype o in o ma ion ex ac ed, which is
especially icky when a p oposal i s ex ac s eco ds and hen ex ac s
he a ibu es wi hin hem, bu also when he p oposal is able o deal
wi h nes ed eco ds and a ibu es. Fo ins ance, T ini y (Sleiman &
Co chuelo,2014) is supposed o ex ac eco ds and a ibu es, bo h
wi hin o he eco ds o in isola ion. Howe e , he au ho s did no
men ion how hey compu ed he e ec i eness measu es ega ding he
eco ds. They jus explained how o compu e hem a he a ibu e le el.
Summing up, decla ing how he e ec i eness measu es a e compu ed
ega ding he s uc u e o he alida ion da ase s should be a mus .
Rega ding he ma ching amongs anno a ions and ex ac ions, F e-
i ag (2000) s a ed ha i is commonly assumed ha he ma chings
mus be exac . Howe e , La elli e al. (2008) ecommended ha his
should be made explici because i is no always c ys al clea . We
ag ee wi h hem since making assump ions migh lead o biased and
misleading conclusions. Only wo o he p oposals su eyed (F ei ag,
2000;de Si e & Daelemans,2003) made i explici he kind o
ma ching pe o med; hey bo h iden i ied he p oblem and p o ed
ha he way he ma ching is in e p e ed has a signi ican impac on
p ecision and ecall. In ui i ely, co ec ma chings should be exac , bu
his in e p e a ion migh be e y s ingen , so he c i e ion o be used
should be es ablished acco ding o he goal o he sys em. I p ecision
is e y impo an , hen exac ma ching should be used; i ha ing high
ecall is mo e impo an , chie ly i some pos -p ocessing can be applied
Expe Sys ems Wi h Applica ions 199 (2022) 116700
3
P. Jiménez and R. Co chuelo
o he ex ac ed in o ma ion, hen con ains ma ching should be used.
Un o una ely, i is no clea o us when o e lapping ma ching may be
a sensible choice.
Summing up, he alida ions pe o med in he li e a u e a e e y
di e se and many de ails ha e commonly no been un eiled, which
makes i di icul o de e mine which p oposal ac ually pe o ms be e
han he o he s. ARIEX was de eloped on he hope o help esea che s
alida e hei p oposals (Jiménez e al.,2016), bu an in-dep h analysis
o he li e a u e has e ealed h ee de iciencies ha mo i a ed us o
wo k on his a icle, namely: how comple e he anno a ions in he
alida ion da ase s a e, how he s uc u e o he ex ac ed in o ma ion
is aken in o accoun , and how he ma chings amongs he anno a ions
and he ex ac ions a e compu ed.
3. P elimina ies
In his sec ion, we p esen some p elimina ies ha basically in o-
duce he ocabula y used h oughou he a icle.
De ini ion 1 (No a ion).A mapping om se 𝑋on o se 𝑌is a unc ion
ha es ablishes co espondences be ween he elemen s o bo h se s. 𝑋
is e e ed o as he domain o he mapping and 𝑌as i s ange. We
deno e he se o mappings om 𝑋on o 𝑌as 𝑋↦𝑌; gi en a mapping
𝑚∈𝑋↦𝑌we deno e i s domain as dom 𝑚and i s ange as an 𝑚;
we deno e he ex ension o mapping 𝑚as {𝑥1↦𝑦1, 𝑥2↦𝑦2,…, 𝑥𝑛↦
𝑦𝑛}; he componen s o a mapping a e e e ed o as co espondences.
Wi hou any loss o gene ali y, we assume ha mappings a e implici ly
so ed acco ding o an a bi a y o al p e-o de on hei domains,
e.g., he s anda d lexicog aphic o al p e-o de .
De ini ion 2 (Values).A alue is a s ing o okens. We deno e he
se o all alues as 𝑉; gi en a alue 𝑣∈𝑉, we deno e i s leng h in
okens as |𝑣|. In ou p oposal, we use he ollowing kinds o okens:
wo ds (sequences o le e s, digi s, and dashes), blanks (sequences o
spaces, abula o s, line eeds, ca iage e u ns, and o m eeds), and
o he symbols (punc ua ion symbols, cu ency symbols, ma h symbols,
and he like). Gi en a mapping 𝑚∈𝑋↦𝑌, we deno e i s alue as
𝑚 and de ine i as i s ex ual se ialisa ion; gi en a co espondence 𝑐in
mapping 𝑚, we also deno e i s alue as 𝑐.
De ini ion 3 (Da ase s).An a ibu e is a label ha we use o g oup
alues ha ha e he same seman ics. We deno e he se o all a ibu es
as 𝐴. An ins ance o a eco d is a mapping om 𝐴↦𝑉, whe e each
co espondence is an ins ance o an a ibu e. A da ase is a collec ion
o eco d ins ances ha mus be ex ac ed om a collec ion o docu-
men s (posi i e da ase ) o mus no be ex ac ed om hem (nega i e
da ase ). A alida ion da ase is a uple o he o m (𝑃 , 𝑁), whe e 𝑃
deno es a posi i e da ase and 𝑁a nega i e da ase . A da ase ha
p o ides he eco d ins ances ha ha e been ac ually ex ac ed om a
documen using an in o ma ion ex ac o is e e ed o as an ex ac ed
da ase . The eco d ins ances in a alida ion da ase a e e e ed o as
anno a ions; he eco d ins ances in an ex ac ed da ase a e e e ed
o as ex ac ions. Fo he sake o eadabili y, we assume ha 𝐴can
be decomposed in o subse s 𝐴𝑉, which p o ides he a ibu es used
in alida ion da ase s, and 𝐴𝐸, which p o ides he a ibu es used in
ex ac ed da ase s. Gi en an a ibu e 𝑎∈𝐴and a da ase 𝐷, we deno e
he se o ins ances o 𝑎in 𝐷as ins ances(𝑎, 𝐷).
No e 1. Ou de ini ion equi es da ase s o be composed o la
eco d ins ances, which does no imply any loss o gene ali y. Gi en
an a bi a y eal-wo ld da ase wi h a ibu e ins ances and possibly
nes ed eco d ins ances, one can ans o m i in o ou model as ollows:
e e y eco d-based da ase can be ep esen ed as a ee in which he
nodes a e he a ibu e o he eco d ins ances and he edges ep esen
he con ainmen ela ionships amongs hem; o ans o m i in o ou
model, i su ices o ep esen ha ee as a la collec ion o pa hs
om he op le el eco ds o hei a ibu es; op-le el a ibu es can
be ans o med in o eco ds by g ouping hem in o a ic i ious global
eco d.
De ini ion 4 (Simila i y).We ha e de ised he ollowing simila i y
measu e: sim(𝑣, 𝑤) = 1 − (𝑖+𝑑)∕(𝑖+𝑑+𝑝), whe e 𝑣and 𝑤deno e
wo alues, (𝑖, 𝑑, 𝑝) = di (𝑣, 𝑤), and di deno es he classical di
algo i hm (Hun & McIl oy,1976). When he di algo i hm is applied o
alues 𝑣and 𝑤, i inds he okens ha mus be, espec i ely, inse ed,
dele ed, o p ese ed in alue 𝑣 o ans o m i in o alue 𝑤; we do no
ac ually equi e o know he exac okens o be inse ed, dele ed, o
p ese ed, bu he numbe o such okens, which we deno e as (𝑖, 𝑑, 𝑝).
No e 2. The li e a u e p o ides a a ie y o simila i y measu es (Yu
e al.,2016), each o which was de ised o add ess pa icula p oblems.
We eso ed o he p e ious de ini ion o simila i y because i in e p e s
he concep as he pe cen age o changes ha mus be applied o a
alue in o de o ans o m i in o ano he alue. This is a p oblem-
agnos ic o mula ion ha p o ed o wo k e y well o de e mine he
mappings amongs he anno a ions and he ex ac ions (which a e
gene ally simila , bu no iden ical). Explo ing o he simila i y mea-
su es and de e mining which one maximises he di e ence be ween
he e ec i eness measu es a ained wi h p e ious alida ions and ou
p oposal would ha e biased i .
4. Guideline
In his sec ion, we p esen a guideline ha complemen s he ARIEX
me hod (Jiménez e al.,2016). I desc ibes ou h ee new ecommen-
da ions, which a e aligned wi h he h ee issues ha we ha e iden i ied
ega ding alida ing in o ma ion ex ac o s.
4.1. Comple eness o alida ion da ase s
We ecommend ha esea che s mus epo on he deg ee o com-
ple eness o hei alida ion da ase s.
A alida ion da ase is comple e i i p o ides an anno a ion o
e e y piece o in o ma ion o be ex ac ed. In he con ex o web-
scale, unsupe ised, o heu is ic-based in o ma ion ex ac o s, p oduc-
ing comple e alida ion da ase s is a e y di icul and e o -p one
ask, mainly due o he human e o equi ed o manually anno a e
web documen s and o polish he anno a ions. As a conclusion, we
hink ha ypical web-scale alida ion da ase s a e pa ially anno a ed,
which implies ha con usion ma ices can only be compu ed pa ially;
his, in u n, has an impac on he esul ing e ec i eness measu es.
Fo ins ance, a piece o in o ma ion ha is ex ac ed bu does no
co espond o any anno a ions in he alida ion da ase canno be
compu ed as a alse posi i e because i is no possible o disce n i i
mus no ha e been ex ac ed o i i was simply no anno a ed.
4.2. S uc u e o in o ma ion
We ecommend ha esea che s mus make i explici how hey
alida ed he s uc u e o he in o ma ion ex ac ed.
We ha e ound ha esea che s ocus on compu ing e ec i eness
measu es on a pe -a ibu e basis and hen a e age he esul s o com-
pu e pe - eco d measu es, i any. Simply pu , complex eco d s uc u es
a e neglec ed since he in o ma ion is basically deal wi h as i i was
o ganised in o uples wi h simple a ibu es. As a conclusion, he abili y
o an in o ma ion ex ac o o ex ac in o ma ion ha is p ope ly
s uc u ed is no aken in o accoun when compu ing he e ec i eness
measu es.
Expe Sys ems Wi h Applica ions 199 (2022) 116700
4
P. Jiménez and R. Co chuelo
4.3. Ma ching ex ac ions and anno a ions
We ecommend ha esea che s mus make i explici how hey
compu ed he ma ches amongs anno a ions and ex ac ions.
The s anda d in he li e a u e is ha ma ches a e compu ed as
ue posi i es, unma ched ex ac ions a e compu ed as alse posi i es,
unma ched anno a ions a e compu ed as alse nega i es, and ue neg-
a i es a e compu ed om he anno a ions ha explici ly e e o pieces
o in o ma ion no o be ex ac ed. The p oblem is ega ding he de -
ini ion o ma ching (La elli e al.,2008), namely: (a) exac ma ching,
which equi es he anno a ed and he ex ac ed alues o be exac ly
he same o be conside ed a ma ching; (b) con ains ma ching, which
equi es he anno a ed alue o con ain he ex ac ed alue; and (c)
o e lapping, which equi es he anno a ed and he ex ac ed alues
o ha e some okens in common. We ecommend ha pa ial ma ches
mus con ibu e o he coun e o ue posi i es ( he pa o an ex ac-
ion ha coincides wi h i s co esponding anno a ion), he coun e o
alse posi i es ( he pa o he ex ac ion ha is no in he anno a ion),
and he coun e o alse nega i es ( he pa o he anno a ion ha is no
in he ex ac ion).
5. Compu ing e ec i eness
In his sec ion, we p esen ou me hod o compu e a con usion
ma ix, which is he basis o compu e a a ie y o s anda d e ec i eness
measu es (Fe i e al.,2009;Sokolo a & Lapalme,2009), and he
deg ee o spu iousness, which helps unde s and hem be e .
I wo ks on a uple o he o m (𝑃 , 𝑁, 𝐸, 𝑝), whe e (𝑃 , 𝑁)is a
alida ion da ase , 𝐸is an ex ac ed da ase , and 𝑝indica es whe he
he alida ion da ase is comple e o no . I pe o ms he ollowing
s eps: i i s inds a mapping om he a ibu es in he alida ion
da ase on o he a ibu es in he ex ac ed da ase ; i hen e-no malises
he eco d ins ances in he ex ac ed da ase acco ding o he p e ious
mapping; nex , i inds a mapping amongs he eco d ins ances in
he alida ion da ase and he ex ac ed da ase ; inally, i compu es
a con usion ma ix and a spu iousness deg ee.
The ollowing subsec ions desc ibe he de ails behind each s ep.
Two o hem ely on a gene al algo i hm o compu e mappings om
a simila i y ma ix, which is p esen ed a he end o he sec ion.
5.1. Find an a ibu e mapping
The goal is o ind a mapping ha makes i explici he co espon-
dences be ween he a ibu es used in he alida ion da ase and he
a ibu es used in an ex ac ed da ase . This is i ial in cases in which
he in o ma ion ex ac o was lea n so ha i labels he alues ha i
ex ac s wi h he labels in he alida ion da ase , which is ypically he
case o in o ma ion ex ac o s ha a e lea n supe isedly. I is mo e
in ol ed in cases in which he in o ma ion ex ac o uses compu e -
gene a ed labels ha ha e no hing o do wi h he labels used in he
alida ion da ase , which is ypically he case o in o ma ion ex ac o s
ha a e lea n unsupe isedly o a e based on heu is ics.
Basically, we need o compu e a mapping ha assigns a simila i y
sco e in in e al [0.00,1.00] o e e y pai o a ibu es (𝑎, 𝑒), whe e 𝑎is
an a ibu e in he alida ion da ase (𝑃 , 𝑁), i.e., 𝑎∈𝐴𝑉, and 𝑒is an
a ibu e in he ex ac ed da ase , i.e., 𝑒∈𝐴𝐸; gene ally speaking, he
highe he sco e, he mo e simila wo a ibu es a e and ice e sa.
To compu e his a ibu e mapping, we i s need o ind he maximum
simila i y be ween a alue 𝑖o an a ibu e 𝑎 om he alida ion da ase
and any o he alues o a ibu e 𝑒 om he ex ac ion da ase as
ollows:
𝑑(𝑖, 𝑒) = max
𝑗∈ins ances(𝑒,𝐸)
sim(
ı,
ȷ).
We hen compu e he simila i y be ween a ibu es 𝑎and 𝑒as
ollows:
𝑚(𝑎, 𝑒) = a g
𝑖∈ins ances(𝑎,𝑃 ∪𝑁)
𝑑(𝑖, 𝑒).
No e ha mapping 𝑚can be in e p e ed as a simila i y ma ix
because i p o ides a simila i y sco e o e e y pai o a ibu es in he
Ca esian p oduc o 𝐴𝑉and 𝐴𝐸. The cells o his ma ix ep esen he
a e age maximum simila i y be ween he alues o 𝑎and he alues o
𝑒. Compu ing an a ibu e mapping om 𝑚is ela i ely s aigh o wa d,
since we only need o selec he pai s in 𝐴𝑉×𝐴𝐸 ha maximise 𝑚.
To p e en p oducing a mapping o each a ibu e in cases in which
he maximum simila i y is e y small, we in oduce a use -de ined
h eshold 𝜃below which no a ibu e mappings a e accep ed. The
de ails o he ancilla y p ocedu e o compu e he mapping a e p o ided
a he end o he sec ion.
5.2. Re-no malise he ex ac ed eco ds
This s ep consis s in changing he names o he a ibu es in 𝐸
acco ding o he mapping ha we ha e compu ed in he p e ious
s ep. This also equi es o e-so he a ibu e ins ances in he eco ds
acco ding o he o al p e-o de used (by de aul , he lexicog aphic
one). This helps align he anno a ed eco ds and he ex ac ed eco ds,
which acili a es mapping hem in he nex s ep. Fu he mo e, i he
alida ion da ase is no comple e, hen we mus emo e e e y a ibu e
in 𝐸 o which a mapping has no been ound in he p e ious s ep.
The alues o he unmapped a ibu es om 𝐸con ibu e o inc easing
he spu iousness o he esul s i he alida ion da ase is pa ially
anno a ed; o he wise, hey a e coun ed as alse posi i es. A simila
a gumen ollows o he unmapped a ibu es in 𝑃∪𝑁: he unmapped
a ibu es om 𝑃con ibu e o he coun o alse nega i es and he
unmapped a ibu es om 𝑁con ibu e o he coun o ue nega i es.
5.3. Find a eco d mapping
We ely on he same gene ic mapping algo i hm as be o e o com-
pu e he co espondences be ween he eco ds in a alida ion da ase
and an ex ac ed da ase . In his case, we compu e a simila i y ma ix
𝑚as ollows:
𝑚(𝑠, 𝑡) = sim(𝑠, 
𝑡),
o any 𝑠 ∈ (𝑃∪𝑁)and 
𝑡∈𝐸. Fo his de ini ion o wo k well, i is
necessa y ha he eco d ins ances in 𝑃∪𝑁and 𝐸be p ocessed by
means o he p e ious s eps so ha he eco ds a e well aligned be o e
hei simila i y is compu ed.
Once he simila i y ma ix 𝑚is compu ed, we can use he same
gene ic me hod as be o e o compu e he eco d mapping. The de ails
a e p o ided a he end o he sec ion.
5.4. Compu e he con usion ma ix
Gi en a alida ion da ase (𝑃 , 𝑁), an ex ac ed da ase 𝐸, and a
mapping 𝑟amongs hei eco ds, we i s se e e y componen o he
con usion ma ix o ze o and hen i e a e as ollows: (a) o e e y eco d
alue 𝑠in he posi i e da ase ha has been mapped on o a eco d
in he ex ac ed da ase , we compu e 𝑡=𝑟(𝑠),𝑘=𝑖+𝑑+𝑝, and
(𝑖, 𝑑, 𝑝) = di (𝑠, 
𝑡); we hen inc ease he coun o ue posi i es by 𝑝∕𝑘,
i.e., he pe cen age o okens ha ha e been co ec ly ex ac ed, he
coun o alse nega i es by 𝑑∕𝑘, i.e., he pe cen age o okens ha a e
in he anno a ion bu ha e no been ex ac ed, and he coun o alse
posi i es by 𝑖∕𝑘, i.e., he pe cen age o okens ha ha e been ex ac ed
bu do no co espond o any okens in he anno a ion. (b) Fo e e y
eco d alue 𝑠in he posi i e da ase ha has no been mapped on o a
eco d in he ex ac ed da ase , we inc ease he coun o alse nega i es
by one. (c) Fo e e y eco d alue in he nega i e da ase ha has
been mapped on o a eco d alue in he ex ac ed da ase , we inc ease
he coun o alse posi i es by one. (d) Fo e e y eco d alue in he
nega i e da ase ha has no been mapped on o any eco d alues in
Expe Sys ems Wi h Applica ions 199 (2022) 116700
5
P. Jiménez and R. Co chuelo
Table 1
Desc ip ion o he da ase s.
Ca ego y Da ase Schema Docs Size (KiB) Posi i es
Jobs
Insigh in o Di e si y Job{company, loca ion, ca ego y} 30 80 30.00
4 Jobs Job{company, loca ion, ca ego y} 30 80 30.00
6 Figu e Jobs Job{company, loca ion, ca ego y} 30 73 30.00
Ca ee Builde Job{company, loca ion, ca ego y} 30 54 30.00
Job o Mine Job{company, loca ion, ca ego y} 30 24 30.00
Ca s
Au o T ade Ca {colo , doo s, engine, mileage, model, p ice, ansmission, ype} 30 184 30.00
Ca Max Ca {colo , mileage, model, p ice, ansmission, yea , ype} 30 67 30.00
Ca Zone Ca {colo , doo s, engine, loca ion, make, model, p ice, ansmission, yea , ype} 30 71 30.00
Classic Ca s o Sale Ca {colo , loca ion, make, model, p ice, ansmission, yea , ype} 30 76 28.90
In e ne Au oguide Ca {colo , doo s, engine, loca ion, mileage, p ice, ansmission, ype} 30 154 30.00
Books
Abe Books Book{ i le, au ho , p ice, isbn} 30 38 35.60
Awesome Books Book{ i le, au ho , p ice, isbn, yea } 30 20 37.17
Be e Wo ld Books Book{ i le, au ho , p ice} 30 125 34.50
Many Books Book{ i le, au ho , yea } 30 27 30.50
Wa e s ones Book{ i le, au ho , p ice} 30 80 31.50
he ex ac ed da ase , we inc ease he coun o ue nega i es by one.
(e) I he alida ion da ase is comple e, we also inc ease he numbe
o alse posi i es by he coun o eco d alues in he ex ac ed da ase
ha do no co espond o any eco d alues in he posi i e da ase .
F om his con usion ma ix, one can compu e a a ie y o e ec i eness
measu es (Yu e al.,2016).
5.5. Compu e he spu iousness deg ee
I he anno a ion o he alida ion da ase is no comple e, we hen
compu e he spu iousness deg ee as he pe cen age o eco d alues in
he ex ac ed da ase o which he e is no a co espondence in he
posi i e da ase .
5.6. Gene ic me hod o compu e mappings
Gi en a ma ix 𝑚 ha p o ides he simila i y be ween any wo
objec s in he Ca esian p oduc o wo a bi a y se s, he me hod wo ks
as ollows: i i s inds he pai o objec s (𝑝, 𝑞)whose simila i y is
maximum; hen, as long as he ma ix is no emp y and he maximum
simila i y is no smalle han a use -de ined h eshold 𝜃, he algo i hm
maps 𝑝on o 𝑞, emo es ha pai om ma ix 𝑚, and con inues i e a -
ing. Th eshold 𝜃mus be se by he use p io o execu ing ou me hod.
I allows o ine- une how demanding he mapping is: he g ea e his
h eshold, he less mappings a e compu ed and ice e sa.
No e ha ou simila i y measu e e u ns alues in ange [0.00,1.00],
which acili a es in e p e ing 𝜃. Simply pu , 𝜃is he one-complemen
o he pe cen age o changes ha mus be ca ied ou in a alue o
ans o m i in o ano he alue. Fo ins ance, se ing 𝜃= 0.80 means
ha he maximum allowable pe cen age o change o assume ha an
objec can be mapped on o ano he objec is 20%.
6. Expe imen a ion
In his sec ion, we epo on he esul s o ou expe imen al analysis.
Fi s , we p esen he de ails o ou expe imen al se ing and hen
p esen and analyse wo expe imen s ha help us p o e ha ou guide-
line o complemen ARIEX may lead o esul s ha a e signi ican ly
di e en , which p o es ha he h ee issues ha we ha e iden i ied
a e eally impo an .
6.1. Expe imen al se ing
We used a collec ion o 15 da ase s on jobs, ca s, and books ha
we e andomly selec ed om he ARIEX eposi o y (Jiménez e al.,
2016). Table 1 shows a desc ip ion, namely: he columns ep esen
he domains, he si es, he schema o he eco ds and a ibu es ha
we e anno a ed, he numbe o documen s collec ed, hei size in KiB,
and he a e age numbe o posi i e alues anno a ed. No e ha each
documen p o ides a o m wi h in o ma ion abou one i em, ha is,
one eco d mus be ex ac ed om each o hem. When he a e age
numbe o posi i e examples is g ea e han he numbe o documen s,
i means ha he e a e se e al alues o a gi en a ibu e. Con a ily,
i his numbe is smalle , i means ha he e a e some missing alues
o some o he a ibu es. These da ase s we e enough o p o e ha
he decisions made ega ding he h ee issues ha we ha e iden i ied
ha e a signi ican impac on he e ec i eness esul s.
We expe imen ed wi h ou web in o ma ion ex ac o s, namely: (a)
Wien (Kushme ick e al.,1997), which is a classical supe ised p oposal
ha lea ns he delimi e s a ound he in o ma ion o be ex ac ed; (b)
Tango (Jiménez & Co chuelo,2016a), which is a ecen supe ised
p oposal ha lea ns i s -o de ules whose p edica es a e based on i-
sual, s uc u al, use -de ined, and con en -based ea u es; (c) RoadRun-
ne (C escenzi e al.,2001), which is a classical unsupe ised p oposal
ha a emp s o in e he empla e o se e al documen s by compa ing
hei sha ed and non-sha ed okens; and (d) Ho Web (Roldán e al.,
2017),1which is a heu is ic-based p oposal ha a emp s o iden i y
common isual pa e ns o p esen in o ma ion.
We ca ied ou wo expe imen s. The i s one con on ed Wien
and Tango; he goal was o con i m ha he way he ma chings a e
compu ed may ha e a signi ican impac on he esul s. The second one
con on ed RoadRunne and Ho Web; in his case, he emphasis was on
con i ming ha he deg ee o spu iousness ma e s signi ican ly when
compa ing unsupe ised o heu is ic-based in o ma ion ex ac o s. In
bo h cases, we compu ed he s anda d e ec i eness measu es, namely:
p ecision ( a io o ue posi i es o ue posi i es plus alse posi i es),
ecall ( a io o ue posi i es o ue posi i es plus alse nega i es), and
he 𝐹1sco e ( he wo-ha monic a e age o p ecision and ecall).
We pe o med he s a is ical analyses using he Wilcoxon signed-
ank es , which is a non-pa ame ic es o compa e wo popula ions.
In ou case he popula ions co espond o he esul s when applying he
o iginal alida ion p ocedu e and he esul s a ained when applying
he new alida ion p ocedu e, ega ding each o he e ec i eness mea-
su es. I he esul ing 𝑝- alue is smalle han he s anda d signi icance
le el (𝛼= 0.05), he di e ences a e signi ican , which demons a es
ha he impac o he h ee issues unde s udy may ac ually bias he
conclusions.
6.2. Expe imen #1
Fi s , we expe imen ed wi h Wien and Tango, which a e supe ised.
The esul s o his expe imen a e shown in Table 2. The i s wo
1Roldán e al. (2017) did no use a speci ic name o e e o hei p oposal;
we ha e dubbed i Ho Web a e he name o he con e ence whe e i was
p esen ed o acili a e e e encing i .

Expe Sys ems Wi h Applica ions 199 (2022) 116700
6
P. Jiménez and R. Co chuelo
Table 2
Resul s o Expe imen #1.
Ca ego y/Da ase Wien O iginal alida ion Wien New alida ion Tango O iginal alida ion Tango New alida ion
P R 𝐹1P R 𝐹1P R 𝐹1P R 𝐹1
Jobs
Insigh in o Di e si y 1.00 0.33 0.50 1.00 0.39 0.56 0.97 0.92 0.94 0.77 0.92 0.84
4 Jobs 1.00 1.00 1.00 0.95 0.99 0.97 0.97 0.78 0.87 0.93 0.78 0.85
6 Figu e Jobs 1.00 1.00 1.00 0.97 0.99 0.98 0.82 0.93 0.87 0.82 0.93 0.87
Ca ee Builde 1.00 0.33 0.50 0.02 0.05 0.08 0.93 0.89 0.91 0.79 0.84 0.81
Job o Mine 1.00 0.67 0.80 1.00 0.63 0.77 0.99 0.96 0.97 0.96 1.00 0.98
Ca s
Au o T ade 1.00 1.00 1.00 1.00 1.00 1.00 0.96 0.96 0.96 0.92 0.96 0.94
Ca Max 1.00 0.86 0.92 0.94 0.94 0.94 1.00 1.00 1.00 0.99 0.99 0.99
Ca Zone 1.00 0.87 0.93 0.82 0.69 0.75 0.95 0.97 0.96 0.94 0.97 0.95
Classic Ca s o Sale - - - - - - 0.91 0.97 0.94 0.91 0.97 0.94
In e ne Au oguide - - - - - - 0.92 0.95 0.94 0.89 0.95 0.92
Books
Abe Books 1.00 1.00 1.00 0.99 0.92 0.95 1.00 0.94 0.97 0.78 0.94 0.85
Awesome Books 1.00 0.67 0.80 1.00 0.67 0.80 0.99 1.00 0.99 0.83 0.99 0.90
Be e Wo ld Books 1.00 0.01 0.17 0.01 0.00 0.00 1.00 1.00 1.00 0.66 0.95 0.78
Many Books 1.00 0.96 0.98 0.99 0.96 0.98 0.97 0.98 0.97 1.00 0.99 0.99
Wa e s ones 1.00 0.92 0.96 1.00 0.91 0.96 1.00 1.00 1.00 0.95 1.00 0.97
A e age 1.00 0.74 0.81 0.82 0.70 0.75 0.96 0.95 0.95 0.88 0.95 0.91
S anda d de ia ion 0.00 0.32 0.26 0.36 0.35 0.34 0.05 0.06 0.04 0.10 0.06 0.07
columns epo on he ca ego y and he da ase s, and he emaining
ones epo on he e ec i eness measu es compu ed o each p oposal
on hose da ase s in he con ex o he o iginal alida ion and he new
alida ion pe o med in his expe imen . The cells wi h a dash indica e
ha he p oposal in he co esponding column was unable o ex ac
in o ma ion om he da ase in he co esponding ow. The las wo
ows epo on he a e age and he s anda d de ia ion o he measu es.
Rega ding he comple eness o he alida ion da ase s, i was 100%
in his case because he da ase s we e comple ely anno a ed wi h he
exac in o ma ion o be ex ac ed. Tha means ha e e y piece o
in o ma ion ex ac ed ha was no mapped was coun ed as a alse
posi i e in he con usion ma ix. Rega ding how he in o ma ion is
s uc u ed, he anno a ed da ase s ha e one le el o nes ing: i s he
eco d alues we e anno a ed and hen he a ibu e alues we e anno-
a ed wi hin hem. Thus, we compu ed he esul s as an a e age o hei
a ibu e and eco ds alues. I a eco d was no ex ac ed o inco ec ly
ex ac ed, hei co esponding a ibu e alues we e assumed no o be
ex ac ed and hei anno a ions we e he e o e coun ed as alse nega-
i es. Rega ding how ma ches we e compu ed, he o iginal alida ions
used con ains ma chings; he new alida ion used exac ma chings,
which a e mo e s ingen and app op ia e o supe ised p oposals
since hey a e in ended o lea n he exac pieces o in o ma ion o
ex ac om he anno a ions. The use -de ined h eshold 𝜃 o ind he
a ibu e o eco d mappings was se o 80% in bo h cases.
The esul s in Table 2 make i clea ha he e a e di e ences in
he e ec i eness be ween he o iginal and he new e alua ion. The
esul s a e wo se in he new alida ion, which makes i clea ha using
exac ma chings is mo e s ingen . I has a clea nega i e impac on he
p ecision and he 𝐹1sco e, e en hough he ecall is simila . In bo h
p oposals, he e a e h ee equen si ua ions, namely: he p ecision
ge s wo se, which occu s when he e a e mo e ex ac ed okens han
anno a ed ones because hey coun as alse posi i es; he ecall ge s
be e , which happens when he numbe o okens ha ha e no been
ex ac ed is a small ac ion o he o al numbe o okens in he eco d
so ha hey coun as ac ional numbe s ins ead o whole numbe s; he
ecall ge s wo se, which is he opposi e case.
Now, we need o p o e ha he di e ences a e ac ually signi ican
a he s anda d signi icance le el. The esul s o he s a is ical analysis
a e shown in Table 3. The i s wo columns e e o he e ec i eness
measu es and whe he hey co espond o he o iginal o he new
alida ion; he ollowing columns epo on hei empi ical anks,
minimum and maximum alues, a e age alue and s anda d de ia ion,
and he 𝑝- alue compu ed by he Wilcoxon signed- ank es . No e ha
all o he p- alues a e clea ly smalle han he s anda d signi icance
le el, which is a s ong indica ion ha he di e ences be ween he
esul s in he o iginal alida ion and he new alida ion a e signi ican .
6.3. Expe imen #2
Second, we expe imen ed wi h RoadRunne , which is unsupe ised,
and Ho Web, which is heu is ic-based. Rega ding RoadRunne , we
lea n a empla e om wo andom documen s and hen applied i
o he emaining ones. We epea ed he p ocess wi h andom pai s o
lea ning documen s and we selec ed he bes pe o ming one. We did
no use mo e lea ning documen s because hei a iabili y makes i e y
di icul o ind a common empla e as he numbe o lea ning docu-
men s inc eases and he p oposal ei he ound meaningless empla es
o did no s op in a sensible ime. Rega ding Ho Web, i does no lea n
any ules, bu has buil -in heu is ics ha a e di ec ly applied o he
inpu documen s. Thus, all o he documen s in he da ase s we e used
o alida ion. Al hough he da ase s a e he same as in he p e ious
expe imen , hey we e pa ially anno a ed in his con ex because he
p oposals do no lea n o ex ac he anno a ions, bu ex ac as much
in o ma ion as possible building on he a iabili y ha hey disco e in
he inpu documen s. Consequen ly, hey ypically ex ac much mo e
in o ma ion han expec ed. The use -de ined h eshold 𝜃 o ind he
a ibu e mappings was se o 80% in he case o he Ho Web p oposal
and 20% in he case o RoadRunne . The eason is ha RoadRunne
ex ac ed alues ha we e ypically much la ge han he anno a ed
alues; hus, hei simila i y d opped signi ican ly.
The esul s o his expe imen a e shown in Table 4, which has an
addi ional column called 𝑆𝐷 ha epo s on he deg ee o spu iousness.
(The igu es we e ounded up o wo decimals, which means ha he
cells wi h a 1.00 ac ually ep esen a igu e in be ween 0.995 and
1.000.) The conclusion is ha ei he he da ase s a e no su icien ly
anno a ed, which is some hing common in he con ex o unsupe ised
o heu is ic-based p oposals, o ha he p oposals ex ac much i el-
e an in o ma ion in which we a e no in e es ed. Ou s is he o me
case: only a ew a ibu es pe documen we e anno a ed, bu all o
hei alues we e anno a ed. No e ha he in e p e a ion o he esul s
migh be misleading when he da ase s a e pa ially anno a ed since
a p oposal ha eaches a p ecision and a ecall close o 1.00 wi h a
spu iousness deg ee close o 1.00 migh be wo se han ano he p oposal
ha eaches a p ecision and a ecall be ween 0.80 and 0.90 wi h a
spu iousness deg ee close o 0.30. The spu iousness deg ee is compu ed
as he a e age o spu ious in o ma ion in e e y eco d wi hin he
alida ion da ase s.
Acco ding o he esul s, RoadRunne pe o ms e y poo ly in ou
alida ion da ase s because i equen ly ex ac ed e y long exce p s
ha con ain he a ibu e alues and much i ele an in o ma ion; in
he new alida ion, his con ibu ed o wo sening he e ec i eness
measu es, which makes he p oblem mo e e iden . I also ends o
Expe Sys ems Wi h Applica ions 199 (2022) 116700
7
P. Jiménez and R. Co chuelo
Table 3
S a is ical analysis o Expe imen #1.
Measu e Valida ion Rank Min Max Mean S de P- alue
O iginal 1.19 1.00 1.00 1.00 0.00
PNew 1.81 0.01 1.00 0.82 0.36 3.96E-03
O iginal 1.27 0.01 1.00 0.74 0.32
RNew 1.73 0.00 1.00 0.70 0.35 5.40E-02
O iginal 1.31 0.17 1.00 0.81 0.26
F1 New 1.69 0.00 1.00 0.75 0.34 3.74E-02
(a) Resul s ega ding Wien.
Measu e Valida ion Rank Min Max Mean S de P- alue
O iginal 1.07 0.82 1.00 0.96 0.05
PNew 1.93 0.66 1.00 0.88 0.10 9.83E-04
O iginal 1.57 0.78 1.00 0.95 0.06
RNew 1.43 0.78 1.00 0.95 0.06 4.44E-01
O iginal 1.20 0.87 1.00 0.95 0.04
PNew 1.80 0.78 0.99 0.91 0.07 5.30E-03
(b) Resul s ega ding Tango.
Table 4
Resul s o Expe imen #2.
Da ase RoadRunne O iginal alida ion RoadRunne New alida ion Ho Web O iginal alida ion Ho Web New alida ion
P R 𝐹1P R 𝐹1S SD P R 𝐹1P R 𝐹1S SD
Jobs
Insigh in o Di e si y 1.00 0.08 0.15 0.19 0.02 0.04 0.14 0.38 0.93 0.50 0.65 0.93 0.50 0.65 0.97 0.03
4 Jobs 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.82 0.90 0.98 0.82 0.90 1.00 0.00
6 Figu e Jobs 1.00 0.29 0.44 0.43 0.17 0.24 0.46 0.50 0.82 0.96 0.88 0.82 0.96 0.88 0.99 0.00
Ca ee Builde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.16 0.27 0.99 0.16 0.27 1.00 0.00
Job o Mine 1.00 0.46 0.63 0.88 0.47 0.61 0.70 0.46 1.00 1.00 1.00 1.00 1.00 1.00 0.01 0.00
Ca s
Au o T ade - - - - - - - - 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.00
Ca Max 0.82 0.47 0.60 0.82 0.40 0.54 0.77 0.42 0.96 0.63 0.76 0.96 0.63 0.76 0.99 0.00
Ca Zone 1.00 0.01 0.02 0.00 0.00 0.00 0.00 0.00 0.88 0.92 0.90 0.88 0.92 0.90 0.99 0.00
Classic Ca s o Sale 0.84 0.11 0.20 0.87 0.14 0.25 0.82 0.39 0.90 0.86 0.88 0.90 0.86 0.88 0.99 0.00
In e ne Au oguide 1.00 0.43 0.60 0.10 0.02 0.03 0.38 0.49 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.00
Books
Abe Books 1.00 0.41 0.58 0.94 0.47 0.63 0.81 0.39 0.98 0.92 0.95 0.98 0.92 0.95 0.97 0.01
Awesome Books 1.00 0.80 0.89 0.97 0.70 0.81 0.68 0.47 1.00 0.65 0.79 1.00 0.65 0.79 0.95 0.03
Be e Wo ld Books - - - - - - - - 1.00 0.91 0.95 1.00 0.91 0.95 0.99 0.00
Many Books 0.58 0.37 0.45 0.73 0.22 0.34 0.86 0.35 1.00 0.38 0.56 1.00 0.38 0.56 1.00 0.00
Wa e s ones 1.00 0.33 0.50 0.30 0.13 0.18 0.83 0.37 1.00 0.89 0.94 1.00 0.89 0.94 0.99 0.00
A e age 0.79 0.29 0.39 0.48 0.21 0.28 0.50 0.32 0.96 0.77 0.83 0.96 0.77 0.83 0.92 0.00
S anda d de ia ion 0.37 0.24 0.29 0.40 0.23 0.28 0.35 0.19 0.06 0.26 0.20 0.06 0.26 0.20 0.25 0.01
Table 5
S a is ical analysis o Expe imen #2.
Measu e Valida ion Rank Min Max Mean S de P- alue
O iginal 1.27 0.00 1.00 0.79 0.37
PNew 1.73 0.00 0.97 0.48 0.40 1.80E-02
O iginal 1.31 0.00 0.80 0.29 0.24
RNew 1.69 0.00 0.70 0.21 0.23 1.96E-02
O iginal 1.23 0.00 0.89 0.39 0.29
F1 New 1.77 0.00 0.81 0.28 0.28 1.05E-02
(a) Resul s ega ding RoadRunne .
Measu e Valida ion Rank Min Max Mean S de P- alue
O iginal 1.50 0.82 1.00 0.96 0.06
PNew 1.50 0.82 1.00 0.96 0.06 5.800E-01
O iginal 1.50 0.16 1.00 0.77 0.26
RNew 1.50 0.16 1.00 0.77 0.26 5.00E-01
O iginal 1.50 0.27 1.00 0.83 0.20
F1New 1.50 0.27 1.00 0.83 0.20 5.00E-01
(b) Resul s ega ding Ho Web.
ex ac se e al a ibu es as one single a ibu e ha is no easy o
spli using pos -p ocessing. Ve y o en, oo, his p oposal ex ac s all
o he a ibu es in a ew documen s, bu ails wi h he o he s because
i canno in e a good common empla e, which is he eason why i s
ecall seldom exceeds 0.40–0.50. I seems ha Ho Web p o ides e y
eliable esul s because i s e ec i eness measu es keep almos he same
in he o iginal alida ion and he new alida ion. The au ho s o iginally
compu ed he e ec i eness measu es a he a ibu e le el using exac
ma chings, bu hey did no analyse he amoun o ex ac ions ha
canno be mapped on o anno a ions.
We ha e also conduc ed a s a is ical analysis. The esul s a e shown
in Table 5. No e ha he 𝑝- alue e u ned by he Wilcoxon signed- ank
es is below he s anda d signi icance le el in he case o RoadRunne ,
which clea ly suppo s he idea ha he di e ences in ank be ween
he o iginal and he new alida ion a e s a is ically signi ican . No e,
oo, ha he p- alues coincide wi h he s anda d signi icance le el in
Expe Sys ems Wi h Applica ions 199 (2022) 116700
8
P. Jiménez and R. Co chuelo
he case o Ho Web, which means ha he expe imen does no suppo
he hypo hesis ha he di e ences in ank a e signi ican ega ding his
p oposal. The eason is, basically, ha he o iginal alida ion was as
s ingen as he new one.
7. Conclusions
In his a icle, we ha e iden i ied h ee key issues ega ding he
alida ion o in o ma ion ex ac o s, namely: (a) comple eness o he
alida ion da ase s, which is o u e mos impo ance o pu he e ec-
i eness measu es in a p ope con ex ega ding he deg ee o spu i-
ousness; (b) he s uc u e o he in o ma ion o be alida ed, so ha
one can know i he p oposal is able o ex ac la s uc u es o nes ed
s uc u es, and how good i is in he la e case; (c) he kind o ma ching
selec ed, which makes he conclusions mo e o less s ingen .
The p e ious issues ha e a signi ican impac on he e ec i eness
measu es, which means ha he compa isons migh be he e ogeneous
and un ai i we do no epo on he p e ious issues. We ha e pe -
o med wo expe imen s ega ding he p e ious ideas. In he i s
expe imen , we wo ked wi h wo supe ised p oposals; he esul s
p o ed ha p ecision dec eased when ou p oposal was used, which
had a nega i e impac on he 𝐹1sco e. In he second expe imen , we
wo ked wi h an unsupe ised and a heu is ic-based p oposal; ou em-
pi ical esul s p o ed ha he deg ee o spu iousness can be signi ican
and mus be epo ed. Ou s a is ical analyses con i med ou ideas.
Summing up, we s ongly ecommend ha esea che s should con-
duc he e alua ion o hei p oposal ollowing he guideline p o ided
so ha he esul s hey publish a e easie o compa e ai ly. They also
ha e o ca e ully compa e hei p oposals o hei compe i o s, as long
as he compe i o s ha e also p o ided he esul s ollowing he same
guideline.
CRediT au ho ship con ibu ion s a emen
Pa icia Jiménez: Concep ualisa ion, Me hodology, So wa e, Val-
ida ion, Resou ces, Da a cu a ion, W i ing – o iginal d a , W i ing
– e iew & edi ing. Ra ael Co chuelo: Concep ualisa ion, Me hodol-
ogy, W i ing – o iginal d a , W i ing – e iew & edi ing, Supe ision,
P ojec adminis a ion, Funding acquisi ion.
Decla a ion o compe ing in e es
The au ho s decla e ha hey ha e no known compe ing inan-
cial in e es s o pe sonal ela ionships ha could ha e appea ed o
in luence he wo k epo ed in his pape .
Acknowledgemen s
The au ho s we e pa ially suppo ed by he Spanish R&D p o-
g amme h ough g an s TIN2016-75394-R and PID2020-112540RB-
C44 (MCIN/AEI/ 10.13039/501100011033), as well as he Andalusian
R&D p og amme h ough g an s P18-RT-1060 and US-1381375.
Re e ences
Ál a ez, M., Pan, A., Raposo, J., Bellas, F., & Cacheda, F. (2010). Finding and
ex ac ing da a eco ds om web pages. Signal P ocessing Sys ems,59(1), 123–137.
h p://dx.doi.o g/10.1007/s11265-008-0270-y.
Baumga ne , R., F ölich, O., & Go lob, G. (2007). The Lix o sys ems applica ions in
business in elligence and he Seman ic Web. In ESWC (pp. 16–26). h p://dx.doi.
o g/10.1007/978-3-540-72667-8_3.
Baumga ne , R., Ga e baue , W., & Go lob, G. (2018). Web da a ex ac ion sys em.
In Encyclopedia o da abase sys ems (2nd ed.). Sp inge , h p://dx.doi.o g/10.1007/
978-1-4614-8265-9_1154.
Chang, C.-H., Kayed, M., Gi gis, M. R., & Shaalan, K. F. (2006). A su ey o
web in o ma ion ex ac ion sys ems. IEEE T ansac ions on Knowledge and Da a
Enginee ing,18(10), 1411–1428. h p://dx.doi.o g/10.1109/TKDE.2006.152.
Chincho , N., Hi schman, L., & Lewis, D. D. (1993). E alua ing message unde s anding
sys ems: an analysis o he hi d Message Unde s anding Con e ence (MUC-3).
Compu a ional Linguis ics,19(3), 409–449.
C escenzi, V., Mecca, G., & Me ialdo, P. (2001). RoadRunne : owa ds au oma ic da a
ex ac ion om la ge web si es. In VLDB (pp. 109–118). URL h p://www. ldb.
o g/con /2001/P109.pd .
Fe a a, E., Meo, P. D., Fiuma a, G., & Baumga ne , R. (2014). Web da a ex ac ion,
applica ions and echniques: a su ey. Knowledge Based Sys em,70, 301–323. h p:
//dx.doi.o g/10.1016/j.knosys.2014.07.007.
Fe i, C., He nández-O allo, J., & Mod oiu, R. (2009). An expe imen al compa ison o
pe o mance measu es o classi ica ion. Pa e n Recogni ion Le e s,30(1), 27–38.
h p://dx.doi.o g/10.1016/j.pa ec.2008.08.010.
F ei ag, D. (2000). Machine lea ning o in o ma ion ex ac ion in in o mal domains.
Machine Lea ning,39(2/3), 169–202. h p://dx.doi.o g/10.1023/A:1007601113994.
Hi schman, L. (1998). The e olu ion o e alua ion: lessons om he Message Un-
de s anding Con e ences. Compu e Speech and Language,12(4), 281–305. h p:
//dx.doi.o g/10.1006/csla.1998.0102.
Hogue, A. W., & Ka ge , D. R. (2005). Th eshe : au oma ing he unw apping o seman ic
con en om he Wo ld Wide Web. In WWW (pp. 86–95). h p://dx.doi.o g/10.
1145/1060745.1060762.
Hsu, C.-N., & Dung, M.-T. (1998). Gene a ing ini e-s a e ansduce s o semi-s uc u ed
da a ex ac ion om he Web. In o ma ion Sys ems,23(8), 521–538. h p://dx.doi.
o g/10.1016/S0306-4379(98)00027-1.
Hun , J. W., & McIl oy, M. D. (1976). Compu ing Science Technical Repo ,An algo i hm
o di e en ial ile compa ison:Technical epo , Bell Labo a o ies, URL h p://www.
cs.da mou h.edu/~doug/di .pd .
I eson, N., Ci a egna, F., Cali , M. E., F ei ag, D., Kushme ick, N., & La elli, A. (2005).
E alua ing machine lea ning o in o ma ion ex ac ion. In ICML, ol. 119 (pp.
345–352). h p://dx.doi.o g/10.1145/1102351.1102395.
I mak, U., & Suel, T. (2006). In e ac i e w appe gene a ion wi h minimal use e o .
In WWW (pp. 553–563). h p://dx.doi.o g/10.1145/1135777.1135859.
Jiménez, P., & Co chuelo, R. (2016a). On lea ning web in o ma ion ex ac ion ules
wi h Tango. In o ma ion Sys ems,62, 74–103. h p://dx.doi.o g/10.1016/j.is.2016.
05.003.
Jiménez, P., & Co chuelo, R. (2016b). Rolle : a no el app oach o web in o ma ion
ex ac ion. Knowledge and In o ma ion Sys ems,49(1), 197–241. h p://dx.doi.o g/
10.1007/s10115-016-0921-4.
Jiménez, P., Co chuelo, R., & Sleiman, H. A. (2016). ARIEX: au oma ed anking o
in o ma ion ex ac o s. Knowledge Based Sys em,93, 84–108. h p://dx.doi.o g/10.
1016/j.knosys.2015.11.004.
Jiménez, P., Roldán, J. C., & Co chuelo, R. (2021). A clus e ing app oach o ex ac
da a om HTML ables. In o ma ion P ocessing & Managemen ,58(6), A icle 102683.
h p://dx.doi.o g/10.1016/j.ipm.2021.102683.
Jiménez, P., Roldán, J. C., Gallego, F. O., & Co chuelo, R. (2020). On he syn hesis o
me ada a ags o HTML iles. So wa e - P ac ice and Expe ience,50(12), 2169–2192.
h p://dx.doi.o g/10.1002/spe.2886.
Kayed, M., & Chang, C.-H. (2010). FiVaTech: page-le el web da a ex ac ion om
empla e pages. IEEE T ansac ions on Knowledge and Da a Enginee ing,22(2),
249–263. h p://dx.doi.o g/10.1109/TKDE.2009.82.
Kushme ick, N., Weld, D. S., & Doo enbos, R. B. (1997). W appe induc ion o
in o ma ion ex ac ion. In IJCAI (1) (pp. 729–737).
La elli, A., Cali , M. E., Ci a egna, F., F ei ag, D., Giuliano, C., Kushme ick, N., &
Romano, L. (2004). A c i ical su ey o he me hodology o IE e alua ion. In
LREC. URL h p://www.l ec-con .o g/p oceedings/l ec2004/summa ies/416.h m.
La elli, A., Cali , M. E., Ci a egna, F., F ei ag, D., Giuliano, C., Kushme ick, N.,
Romano, L., & I eson, N. (2008). E alua ion o machine lea ning-based in o ma-
ion ex ac ion algo i hms: c i icisms and ecommenda ions. Langage Resou ce and
E alua ion,42(4), 361–393. h p://dx.doi.o g/10.1007/s10579-008-9079-3.
Lehne , W. G., & Sundheim, B. (1991). A pe o mance e alua ion o ex -analysis
echnologies. AI Magazine,12(3), 81–94. h p://dx.doi.o g/10.1609/aimag. 12i3.
905.
Pa k, J., & Ba bosa, D. (2007). Adap i e eco d ex ac ion om web pages. In WWW
(pp. 1335–1336). h p://dx.doi.o g/10.1145/1242572.1242838.
Raposo, J., Pan, A., Ál a ez, M., Hidalgo, J., & na, A. V. (2002). The Wa go sys em:
semi-au oma ic w appe gene a ion in p esence o complex da a access modes. In
DEXA Wo kshops (pp. 313–320). h p://dx.doi.o g/10.1109/DEXA.2002.1045916.
Roldán, J. C., Jiménez, P., & Co chuelo, R. (2017). Ex ac ing web in o ma ion
using ep esen a ion pa e ns. In Ho Web (pp. 4:1–4:5). h p://dx.doi.o g/10.1145/
3132465.3133840.
Roldán, J. C., Jiménez, P., & Co chuelo, R. (2020). On ex ac ing da a om ables
ha a e encoded using HTML. Knowledge Based Sys em,190, A icle 105157.
h p://dx.doi.o g/10.1016/j.knosys.2019.105157.
Roldán, J. C., Jiménez, P., Szekely, P., & Co chuelo, R. (2021). TOMATE: a heu is ic-
based app oach o ex ac da a om HTML ables. In o ma ion Sciences,577, 49–68.
h p://dx.doi.o g/10.1016/j.ins.2021.04.087.
Sahugue , A., & Aza an , F. (2001). Building in elligen web applica ions using
ligh weigh w appe s. Da a & Knowledge Enginee ing,36(3), 283–316. h p://dx.
doi.o g/10.1016/S0169-023X(00)00051-3.
Shen, Y. K., & Ka ge , D. R. (2007). U-REST: an unsupe ised eco d ex ac ion sys em.
In WWW (pp. 1347–1348). h p://dx.doi.o g/10.1145/1242572.1242844.
Expe Sys ems Wi h Applica ions 199 (2022) 116700
9
P. Jiménez and R. Co chuelo
de Si e , A., & Daelemans, W. (2003). In o ma ion ex ac ion ia double classi ica ion.
In ATEM Wo kshop (ECML/PKDD) (pp. 1–8).
Sleiman, H. A., & Co chuelo, R. (2013). A su ey on egion ex ac o s om web
documen s. IEEE T ansac ions on Knowledge and Da a Enginee ing,25(9), 1960–1981.
h p://dx.doi.o g/10.1109/TKDE.2012.135.
Sleiman, H. A., & Co chuelo, R. (2014). T ini y: on using ina y ees o unsupe ised
web da a ex ac ion. IEEE T ansac ions on Knowledge and Da a Enginee ing,26(6),
1544–1556. h p://dx.doi.o g/10.1109/TKDE.2013.161.
Sokolo a, M., & Lapalme, G. (2009). A sys ema ic analysis o pe o mance measu es
o classi ica ion asks. In o ma ion P ocessing & Managemen ,45(4), 427–437. h p:
//dx.doi.o g/10.1016/j.ipm.2009.03.002.
Tu mo, J., Ageno, A., & Ca alà, N. (2006). Adap i e in o ma ion ex ac ion. ACM
Compu ing Su eys,38(2), h p://dx.doi.o g/10.1145/1132956.1132957.
Yu, M., Li, G., Deng, D., & Feng, J. (2016). S ing simila i y sea ch and join: a su ey.
F on ie s o Compu e Science,10(3), 399–417. h p://dx.doi.o g/10.1007/s11704-
015-5900-5.
Zhai, Y., & Liu, B. (2005). Web da a ex ac ion based on pa ial ee alignmen . In
WWW (pp. 76–85). h p://dx.doi.o g/10.1145/1060745.1060761.
Zhang, C., Xu, W., Ma, Z., Gao, S., Li, Q., & Guo, J. (2015). Cons uc ion o seman ic
boo s apping models o ela ion ex ac ion. Knowledge Based Sys em,83, 128–137.
h p://dx.doi.o g/10.1016/j.knosys.2015.03.017.