Conside ing he many heo e ical and me hodological iewpoin s in he ield o e alua ion,
a guideline was es ablished o acili a e he e alua ion o social in e en ion p og ams.
Fo his pu pose, he goals o he e alua ion we e aken in o accoun : (a) planning
in e en ions, (b) lea ning and con inuous imp o emen o in e en ions, (c) p og amming
policies, and (d) ans o ma ion o socie y. These goals will de e mine he pe spec i e o
he analysis selec ed ( ocusing on he e aluand o he con ex ) and he s a egies o
change employed ( ocusing on p ocesses o esul s). The elemen s ha , acco ding o
Shadish, Cook, and Le in on (1991), cons i u e he heo y o e alua ion (e aluand, alue,
building o knowledge, and uses) ha e also been conside ed. The analysis o all hese
componen s led o he elabo a ion o a guideline o o ien he p ac ice o e alua ion om
a p agma ic pe spec i e, in acco dance wi h he demands and needs o a ce ain social
con ex .
Keywo ds: p og am e alua ion, social in e en ion, e alua ion heo y, e alua ion p ac ice
Conside ando la mul iplicidad de pe spec i as eó icas y me odológicas exis en es en el
campo de la e aluación, se ha elabo ado una pau a que pe mi a conduci la e aluación
de p og amas de in e ención social. Pa a ello se han enido en cuen a los p opósi os
u obje i os que puede pe segui la e aluación: (a) plani icación de in e enciones, (b)
ap endizaje y mejo a con inua de las in e enciones, (c) p og amación de polí icas, y (d)
ans o mación de la sociedad. Es os obje i os de e mina án la pe spec i a de análisis
adop ada (cen ada en el e aluando o en su con ex o) y las es a egias de cambio
u ilizadas (o ien adas a los p ocesos o a los esul ados). Se han conside ado ambién
los elemen os que pa a Shadish, Cook y Le in on (1991) cons i uyen la eo ía de la
e aluación (e aluando, alo , cons ucción del conocimien o y usos). El análisis de odos
es os componen es ha pe mi ido elabo a una guía que o ien e la p ác ica de la e aluación
desde una pe spec i a p agmá ica aco de a las demandas y necesidades que se plan een
en un con ex o social de e minado.
Palab as cla e: e aluación de p og amas, in e ención social, eo ía de la e aluación,
p ác ica de la e aluación
A P agma ic Guideline o E alua ion o Social In e en ion
Isabel Ma ía He e a Sánchez, José Ma ía León Rubio, and Sil ia Medina Anzano
Uni e si y o Se ille
The Spanish Jou nal o Psychology Copy igh 2005 by The Spanish Jou nal o Psychology
2005, Vol. 8, No. 1, 21-29 1138-7416
Co espondence should be add essed o: Isabel Ma ía He e a Sánchez. Depa amen o de Psicología Social. Uni e sidad de Se illa.
Camilo José Cela s/n. 41018 Se illa (Spain). Phone: 95455988. E-mail: [email p o ec ed]
T ansla ion: Vi ginia Na ascúes Howa d.
21
The pu pose o his wo k is o p o ide a guideline o
o ien e alua o s abou he bes way o pe o m an e alua ion
in he a ea o social in e en ion, depending on he goals,
as hese de e mine how he main e alua ion elemen s a e
se up. Howe e , be o e going in o de ails and o make ou
p oposal mo e comp ehensible, we shall de ine social
in e en ion as a se ies o planned s a egies o achie e
changes in he social eali y in a ce ain popula ion, so ha
his popula ion will each highe le els o de elopmen and
well-being. Thus, he e alua ion o any social in e en ion
p og am in ol es acqui ing knowledge abou he alue o
he p og am and ans e ing such a judgmen o sol ing
social p oblems.
The a ionale o ou p oposal is based on e alua o s’
di icul y in choosing he e alua ion p ocedu e ha bes
ma ches hei demands and needs, gi en he quan i y o
a ailable s a egies and hei di e si y (e.g., Chelimsky &
Shadish, 1997; S u lebaum, 2001).
A esea che ’s i s ask is o ansla e eme ging demands
and needs in o goals. These goals can be g ouped in o one
o he ollowing ca ego ies: (a) planning in e en ions, (b)
lea ning and con inuous imp o emen o in e en ions, (c)
p og amming policies, and (d) ans o ma ion o socie y.
Inclusion in any ca ego y will de e mine he analysis ou look
(mic o o mac o) assumed by he esea che and he change
s a egy adop ed (linea o nonlinea ).
The “mic o” pe spec i e ocuses on s uc u al ( esou ces,
pe sonnel, e c.) and ope a ional (ac i i ies, s a egies, e c.)
cha ac e is ics o he in e en ion p og am ha is he objec
o e alua ion, in o de o de e mine which aspec s a e
imp o able. The “mac o” pe spec i e add esses he poli ical,
economic, o ganiza ional, o psychosocial aspec s o a
p og am (see Shadish, 1987).
Change s a egies e e o he ime pe iod in which he
e alua ion akes place (a he end o du ing he en i e
p ocess) and o he ype o e alua ion pe o med (p ocess-
o ou come- ocused). Acco ding o Veney and Kaluzny
(1991), linea s a egies lead o punc ual e alua ions
pe o med o achie e conc e e goals ha a e usually aimed
a p esen ing accoun s and inc easing he e icacy o impac
o social p og ams. They a e he e o e ypically pe o med
a he end o he in e en ions. Nonlinea s a egies conside
e alua ion and in e en ion o be pa o he same p ocess
ha goes om he in e en ion o he e alua ion, and back
again, so ha a con inued e alua ion o he implici p ocesses
o he p og am is pe o med in o de o in oduce he
pe inen changes o op imize he in e en ion.
Ob iously, e alua ions may ha e a ious goals so ha
in one e alua ion, se e al di e en pe spec i es o analysis
and change s a egies may be adop ed. Fo example, an
e alua ion may p opose o: (a) e i y whe he he
p og ammed ac i i ies a e being adequa ely implemen ed
o achie e con inuous imp o emen (mic o pe spec i e,
nonlinea change); (b) de e mine he e icacy o he p og am
so as o decide whe he o con inue i , modi y i , o elimina e
i (mic o pe spec i e, linea change); (c) e alua e he
di e en ial e ec o di e se p og ams wi h simila
cha ac e is ics on di e en se ings (mac o pe spec i e, linea
change); and (d) assess he ex en o which a ious p og ams
ha a e being ca ied ou p omo e democ a ic alues such
as solida i y, ole ance, and espec o human igh s (mac o
pe spec i e, nonlinea change).
In sho , he goals o an e alua ion de e mine he analysis
pe spec i e adop ed and he way he e aluand is expec ed
o change. Howe e , i s ill emains o be seen how o
pe o m he e alua ion o achie e he p oposed goals. This
leads o he examina ion o he elemen s ha make up he
e alua ion, as hey will de e mine he decisions o be made.
Fo his pu pose, we chose he e alua ion heo y
o mula ed by Shadish, Cook, and Le i on (1991), which
we hink is one o he mos solid and comple e heo e ical
o mula ions. I compiles he con ibu ions o he chie
e alua ion heo e icians o he imes—Sc i en, Campbell,
Weiss, Wholey, S ake, C onbach, and Rossi—conside ing
he ollowing elemen s: (a) he e aluand’s cha ac e is ics;
(b) he e alua ion c i e ia, he pa icipa ion le el o he
g oups in ol ed in he e alua ion, he s andpoin ,
esponsibili y and ole assumed by he e alua o ; (c) he
e alua o ’s scien i ic assump ions and he me hods used o
ob ain in o ma ion; and (d) he use o in luence o he
in o ma ion p o ided by he e alua ion.
The examina ion o each one o hese componen s will
allow us o decide wha o do and how o ac ually conduc
he e alua ion depending on he goals pu sued. We ha e
adop ed a p agma ic s andpoin o he analysis, as we
belie e ha all he op ions a e legi ima e as long as hey
sa is y he needs encoun e ed by he e alua o in he di e en
se ings in which he e alua ion akes place.
The E aluand’s Cha ac e is ics
The sys ema iza ion o he e alua ion p ocess in ol es
conside ing, i s , he objec o he e alua ion—o e aluand—,
which, in ou case, a e he social in e en ion p og ams. We
mus he e o e de e mine he main assump ions and
mechanisms unde lying he in e en ion: he componen s
(assigned esou ces, in e en ion s a egies and ac ics, esul s),
he expec ed causal ela ions among hem, and he con ex ual
ac o s ha accompany hei implemen a ion ( alues, social
no ms, ela ion wi h o he sociopoli ical ac ions, p essu e
om a ious s akeholde s, e c.).
The so-called heo y-d i en e alua ion is a model ha
has ocused mos di ec ly on knowledge o he e aluand.
This mo emen eme ged a he end o he eigh ies as an
al e na i e o he expe imen al models o black box
e alua ion (Chen, 1990), because o i s p oblema ical
applica ion o he e alua ion o social and communi y
in e en ions. This heo y is based on he no ion ha he
design o an e alua ion should es on he logical model
HERRERA, LEÓN, AND MEDINA
22
ha di ec s he p og am o in e en ion i is mean o
e alua e. This means aking in o conside a ion he p ocesses
in ol ed in o mula ing he implemen a ion o he
in e en ion, as well as he esul s expec ed o be achie ed
by he in e en ion.
The app aisal o he p ocesses chie ly in ol es
de e mining whe he he heo y unde lying he p og am
allows one o p opose he desi ed goals and whe he he
implemen a ion o he p og am has been ca ied ou
acco ding o plan. The wo issues ha he no ma i e heo y
(Chen, 1990), he implemen a ion heo y (Weiss, 1998), and
he p ocess heo y (Rossi, F eeman, & Lipsey, 1999) a e
expec ed o cla i y a e he heo y’s su iciency and he
p og am’s adap a ion o i s concep ion
In he case o he e alua ion o he esul s, he emphasis
is on iden i ying he causal mechanisms ha media e be ween
he componen s o a p og am and he esul s. The issues ha
conce n he causa i e heo y (Chen, 1990), he p og am heo y
(Weiss, 1998), and he impac heo y (Rossi e al., 1999) a e
he e icacy, e iciency, and e ec i eness o he p og am.
No doub , his is he bes ype o e alua ion o in es iga e
he concep ion o a p og am and o ga he exhaus i e
knowledge o he e aluand. Howe e , i also p esen s a
se ies o limi a ions, among hem: (a) lack o p ecision in
he elabo a ion o he logical models ha guide he p og am,
among o he easons, because he same p og am may be
based on se e al implici heo ies; o he e alua o , who
may no be in he bes si ua ion o conside all he con ex ual
ac o s o he p og am, may neglec ele an a iables when
concei ing he p og am (Bickman, 2000; Bi ckmaye &
Weiss, 2000); (b) he models esul ing om he esea ch
p ocess pe o med by he e alua o a e so complex ha i
may be impossible o ansla e hem in o a da a analysis
design (Weiss, 1997); and (c) knowledge o he concep ion
o a p og am does no exemp esea che s om using
expe imen al designs o con i m i (Cook, 2000).
Despi e all hese limi a ions, we belie e ha he
elabo a ion o logical models is usually one o he i s asks
o be add essed by he e alua o . Howe e , he p oblem
consis s o de e mining which elemen s o he p og am
should be ope a ionalized wi hin he logical model and hei
impo ance in he se ies o ac i i ies ha make up an
e alua ion. The highe o lesse deg ee o impo ance o he
logical model will depend on he change s a egy adop ed:
1. Ou come- ocused e alua ion. This may no equi e a
logical model, as i can s udy he e ec s o an
in e en ion on a ce ain popula ion, wi hou he need
o app aising he unde lying mechanisms o he
componen s o he p og am and he esul s achie ed.
2. P ocess- ocused e alua ion. The e alua ion ocuses
on he ope a ionaliza ion o he heo y; he e alua o
should de e mine whe he he p og am begins wi h a
heo e ical analysis o he p oblem, whe he i akes
in o accoun how he a ious g oups in ol ed
(manage s, p og amme s, echnicians, use s, e c.)
concei e he p oblem, o whe he he heo y is
compa ible wi h he sys em o dominan alues,
belie s, and ideologies. Fo example, an e alua ion
may ecommend he need o know whe he a p og am
o i s componen s allow o he e alua ion o he
esul s (e alua ion o he e aluabili y).
3. P ocess- and esul s- ocused e alua ion. An example
is he p oposal o Lipsey and Co d ay (2000) o he
e alua ion o he esul s, which in eg a es an
expe imen al o quasi-expe imen al design wi h
heo y-d i en e alua ion, p o iding mo e p ecise
in o ma ion abou he causal mechanisms o he
changes p oduced by he p og am. Tha is: (a)
obse ed a iabili y a ibu ed o ac o s such as he
implemen a ion o he p og am (o ganiza ion o
ac i i ies de eloped and consis en wi h wha was
planned), he pa icipan s’ cha ac e is ics, he ype o
con ac wi h se ice p o ide s, e c.; and (b) he cause-
e ec sequence be ween p og am componen s and
hei in e ela ions wi h p og am esul s.
E alua ion C i e ia, Le el o Pa icipa ion o he
G oups In ol ed, and he E alua o ’s Role
Judgmen s a e an omnip esen ac i i y in any e alua ion
ask. Judgmen is doub less implici in e alua ion, a leas ,
o a compa a i e na u e. This means one mus explici ly
exp ess he logic used o es ablish he c i e ia ha lead o
he o mula ion o he conclusions o he e alua ion. This
emphasis on assessmen and judgmen o he e aluand is
wha dis inguishes e alua ion om esea ch.
This led Sc i en (1980) o es ablish objec i e e alua ion
c i e ia, de eloping a me a- heo y ha he denomina ed he
Logic o E alua ion. Acco ding o his au ho , he in o ma ion
p o ided by he e alua ion is ela ed o ce ain judgmen s
in ou phases: (a) es ablishmen o he c i e ia o me i ha
de e mine how he objec o he e alua ion will be labeled
and judged as good, use ul, o quali y, e c.; (b) elabo a ion
o s anda ds o no ms o unc ioning ha speci y le els o
deg ees o me i ; (c) collec ing da a abou he unc ioning
o he objec and compa ing i wi h p e iously es ablished
unc ioning s anda ds; and (d) syn hesis o in eg a ion o
he da a wi hin a judgmen o me i o alue.
This logic o e alua ion is conside ed a ool ha is
uni e sally applicable o he p ac ice o e alua ion (S ake
& Migo sky, 1997). A gumen s a ise when choosing he
mos pe inen c i e ia o de e mine he alue o me i o
he e aluand, deciding he app op ia eness o he s anda ds
o no ms o unc ioning, and deciding how o measu e he
le el o unc ioning. Howe e , he bigges p oblems c op
up when a emp ing o o mula e he global judgmen om
he syn hesis o he da a collec ed. Thus, whe eas o some
au ho s, alid and objec i e in e ences abou alues can be
ob ained om he da a (Sc i en, 1994), o o he s, his is
EVALUATION OF SOCIAL INTERVENTION 23
impossible because hey conside he da a o be a p ocess
o social cons uc ion and he e o e, subjec o in e p e a ion
by he people in ol ed (Guba & Lincoln, 1989; S ake, 1983;
S ake & Migo sky, 1997).
This leads us o he conside a ion o whe he he
e alua o should se e some p e-es ablished absolu e alues,
o jus desc ibe he alues o ce ain s akeholde s. In his
sense, om a p esc ip i e app oach, we p opose he p imacy
o uni e sal alues ha ci cumsc ibe he e alua ion. The
au ho s who de end his op ion, o example, House (1980),
begin wi h he p emise ha when e alua o s p o ide
in o ma ion abou ce ain ac ions aimed a sol ing social
p oblems (po e y, acism, iolence, e c.), hey a e
pa icipa ing in a mo al ac ha should con ibu e o jus ice
in socie y in gene al.
Va ious di icul ies a ise i we decide o conside alues
as absolu e. Fi s , alues a e no limi ed o a heo y o
philosophical p oposal, so i is impossible o jus i y one heo y
o e ano he based on alues. Second, in e alua ion, he
common p ac ice is o ha e o deal wi h di e en alues and
in e es s ha co espond o di e en g oups o people in ol ed
in he ac ions being e alua ed (Guba & Lincoln, 1989;
Rebolloso & Rebolloso, 1998; Schwand , 1997). In his sense,
he s andpoin o he majo i y conside s i mo e p ac ical o
de end a desc ip i e app oach: E alua ion is limi ed o
desc ibing o no ing his plu ali y o in e es s, es ablishing
he c i e ia o alue on he basis o wha ce ain s akeholde s
conside o pe cei e o be impo an o mo e jus .
P ecisely, he p esence o a ious s akeholde s ha
pa icipa e in he e alua ion p ocess is one o he main
p oblems, because his plu ali y implies he equen
con e gence o con lic ing alues. This si ua ion o ces one
o make decisions abou he need o conside conjoin ly he
in e es s o all he people who pa icipa e in he ac ion, o
o choose he mos ele an among hem. The e a e se e al
iewpoin s on his issue. So, some e alua ion models, such
as he so-called s akeholde e alua ion, de end he need o
“holis ically” conside he conce ns and needs all he a ec ed
g oups (Vedung, 1993). Con e sely, o he s gi e p io i y o
he alues o ce ain g oups, such as, o example, p og am
and se ice manage s (Wholey, 1983), people who lack
powe and who a e no ep esen ed impa ially in democ a ic
decision making (House & Howe, 2000), o mino i y g oups
and hose who ac in hei name (Me ens, 2003).
Along wi h he conside a ion o he a ious g oups ha
a e in e es ed in he e alua i e p ocess as a sou ce o
in o ma ion, he di e en le els o in ol emen o hese
g oups in he e alua ion ask and he e alua o ’s ole a e
also pa icula ly ele an . The e a e also se e al choices
he e. Thus, in he esponsi e e alua ion model, al hough i
con empla es he pa icipa ion o he g oups in ol ed in he
e alua ion, he e alua o is esponsible o and con ols he
e alua ion (House, 2001). Con e sely, in he cons uc i is
app oach o Guba and Lincoln (1989), bo h e alua o s and
he people in ol ed sha e he esponsibili y in a nego ia ion
iewpoin o he e alua i e p ocess. Las ly, he well-known
empowe men e alua ion (Fe e man, 2001) places he
s akeholde s in he cen e o he e alua ion p ocess, looking
o he sel -de e mina ion o he people who pa icipa e in
he p og am; ha is, hei capaci y o de e mine hei own
li e cou ses. This is a model o pa icipa ion and
ans o ma ion in which he e alua o assumes he ole o
p omo e o he lea ning and change p ocesses.
The inclusion o he g oups in ol ed in he e alua ion
p ocess and he ype o ela ion hey ha e wi h he e alua o
ha e eopened a deba e among heo e icians abou he
s andpoin ha he e alua o should main ain du ing he
e alua ion p ocess. I is a deba e o objec i i y e sus
commi men . Some au ho s s a e ha e alua ion should be
neu al, in his case, he e alua o s examine sys ema ically
and empi ically he alues ha a e ele an o he social
policies and p og ams and, i hey ind disc epancy, hey
y o a i e a a consensus be ween he di e en posi ions
o he g oups (Ma k, 2001). Con e sely, o he au ho s de end
a s andpoin ha is mo e commi ed o a ce ain ype o
e alua ion, ha which ad oca es he speci ic in e es s o
ce ain social g oups, in pa icula , hose lacking powe and
igh s (Me ens, 2003).
In sho , hese p oposals unde sco e he di icul y o
assigning p io i y o he alues ha suppo he e alua ion,
o choosing and in ol ing he s akeholde s, and o
de e mining he e alua o ’s habi ual ole. Once again, i is
no ed ha ma ching he e alua ion o he goals can help
sol e his p oblem. Thus, i he aim is o de e mine he
e icacy o a p og am (mic o pe spec i e, linea change),
he alues will al eady be speci ied because hey a e implici
in he in e en ion goals. The e o e, i is no necessa y o
he g oups in ol ed o pa icipa e in his aspec . Simila ly,
he e alua o ’s ole should be neu al because he esul s o
he e alua ion will depend on he causal in e ence being
unbiased. Howe e , i he goal is o iden i y and de e mine
which p og am ou comes will be ele an o he a ious
g oups in ol ed when subsequen ly designing he
in e en ion (mic o pe spec i e, nonlinea change), hen he
alues should be ag eed on by hese g oups, which equi es
hei maximum pa icipa ion in he in es iga ion p ocess.
The e alua o should hen adop a ole o media o o
nego ia o who collec s in o ma ion abou he di e en needs
and who a emp s o each ag eemen s among he a ious
c i e ia and p io i ies p oposed by he g oups in ol ed.
Scien i ic P oposals and Me hods
The key issue in his sec ion is how o ob ain he
knowledge o judge he e aluand; whe he such knowledge
should be ga he ed using quali a i e me hodology such as
ha employed in idiog aphic disciplines, o whe he i should
be based on he adminis a ion o quan i a i e analysis
p ocedu es employed in nomo he ic disciplines. In o he
HERRERA, LEÓN, AND MEDINA
24
EVALUATION OF SOCIAL INTERVENTION 25
wo ds, he issue is whe he he objec o e alua ion is o
unde s and he e aluand o o o mula e gene al laws,
ansla able o wha Pa on (2001) calls he e ec i e
p inciples o p og amming social in e en ions.
Con a y o ex eme iewpoin s ha ega d bo h
me hodological app oaches as opposi es, we ad oca e a
iewpoin ha does no conside hem dicho omic bu
complemen a y. This ou look s a es ha he de ini ion o
he objec o s udy and he goals o he in es iga ion
de e mine he limi s o choice, bo h o he me hodological
app oach and o he analysis echniques (see, o example,
Cook, 1997). The e o e, al hough he epis emological bases
o hese wo me hodological adi ions a e di e en , he e
is no con adic ion in hei conjoin use in social in es iga ion
(Ál a o, 1995; Al i a, 1983; Reicha d & Cook, 1979). This
pe spec i e has p oduced wo speci ic p oposals in he ield
o e alua ion:
1. The use o a speci ic ype o me hodology and i s
co esponding analysis echniques, which does no
imply he accep ance o a whole se ies o me a-
heo e ical o epis emological p oposals. Fo example,
he u iliza ion- ocused e alua ion (Pa on, 1997), a
model di ec ed a using he e alua ion ou come, so
he choice o possible me hodological al e na i es
(expe imen a ion, quasi-expe imen a ion, s a is ical
designs, case s udies, e c.) will depend on whe he o
no he da a ob ained mee he use s’ needs. Fo
ins ance, i he goal o an in e en ion p og am was
he academic in eg a ion o e hnical mino i y g oups,
i migh be su icien o hose in cha ge o know he
equency o in e ac ions be ween s uden s om hese
g oups and hose om he majo i y g oup; in his
case, s a is ical designs would be he mos app op ia e.
Howe e , i he manage s needed o know he na u e
o such in e ac ions, hen obse a ion in si u and in-
dep h in e iews would be equi ed.
2. The co-exis ence o con as ing me hodological
app oaches and analysis echniques (causal and
in e p e i e in es iga ion) in he same heo e ical
p oposal, as occu s, o example, in he ealis ic
e alua ion (Julnes & Ma k, 1998). This e alua ion
model is based on eme gen ealism, which s a es ha
he e is an ex e nal eali y om which obse able
egula i ies may be ex ac ed, which should be
conside ed wi hin he con ex ual dynamic o complex
open sys ems ha a e obse ed indi ec ly. The e o e,
he alue o he di e en me hods will depend on hei
capaci y o e i y he di e en aspec s o he gene a i e
change mechanisms associa ed wi h he social
p og ams and he di e se aspec s o he con ex
a ec ed by he p og am. An example is he case s udy
p esen ed by Ma k, Felle , and Bu on (1997), in which
he impac o a new sys em o con ac ing pe sonnel
in he Adminis a ion was e alua ed. Fo his pu pose,
a se ies o quali a i e me hods (obse a ions and
in e iews) we e inco po a ed in he amewo k o an
expe imen al design. These quali a i e me hods had
se e al goals, among hem: de elopmen and
subsequen e ining o he measu emen ins umen s
(su eys), alida ion o he esul s ob ained in he
su eys, explana ion o he quan i a i e indings, and
se ing he limi s o hei gene aliza ion. Rega ding
he las poin , in he in e iews wi h he di ec o s and
he new employees, he au ho s cla i ied impo ance
o ac o s such as p o essional ca ego y, sala y, wo k
condi ions, and con ac es ic ions o de e mine
whe he o no i was necessa y o implemen he new
con ac ing sys em.
We ad oca e he p oposal o Pa on (1997) and we
coincide wi h his opinion ha he me hodology should be
selec ed acco ding o he use s’ needs, which a e e lec ed
in he e alua ion goals. To a oid subjec ing he e alua ion
o p i a e in e es s, hese goals should be es ablished as a
esul o wha he use s need o know and he e alua o
conside s mos pe inen , gi en he peculia i ies o he objec
o he e alua ion. Thus, a he s a o he p og am, when
li le is known abou he p oblem and i s solu ions, i is mos
app op ia e o disco e and desc ibe he peculia i ies o he
p oblem ( o example, by case s udies). When a p og am is
su icien ly implemen ed a a local le el o be able o o e
an accoun o i s e icacy, he e alua ion will equi e a causal
o p obabilis ic explana ion (expe imen al, quasi-
expe imen al, o co ela ional designs). I he di e en ial
e ec s o simila p og ams in di e en se ings should be
de e mined so as o ensu e he con inua ion o some poli ical
s a egy, he e alua ion should lead o gene alizable
conclusions (such as he case o he syn hesis o esea ch
o me a-analysis). Las ly, in o de o go beyond he impac
o he p og am and ake in o accoun he alue o a ious
possible al e na i es o sol e he social p oblems, he
e alua ion should be o an in e p e i e na u e, using a
he meneu ic model.
In sho , we belie e ha e alua ion should allow one o
es ablish a se ies o gene al p inciples applicable o any
social in e en ion, wi hou ha ing o gi e up unde s anding
he speci ic e aluand in which he e alua ion o igina ed.
The U iliza ion o In luence o he In o ma ion
P o ided by he E alua ion
Resea ch in his ield has ocused on iden i ying how
he indings o an e alua ion a ec he di e en s akeholde s.
Ini ially, he implici use heo ies s a ed ha he esul s o
an e alua ion should become he cen al inpu s in poli ical
decision making (ins umen al use). Howe e , expe ience
soon showed he e alua o s ha poli ical decisions we e no
made acco ding o he esul s o he e alua ion; some imes
because he esul s we e con a y o poli ical in e es s and
o he s because hey we e no de ini e.

HERRERA, LEÓN, AND MEDINA
26
Al hough he esul s do no lead o new ac ions o o
he modi ica ion o he e alua ed ac ions, hey may be used
o such ends as: (a) o e iew he in e en ion ounda ion,
indica ing possible di ec ions o ac ion, which Le i on and
Hughes (1981) called concep ual use, and C onbach (1982)
and Weiss and Bucu alas (1981) enligh enmen ; and (b) o
s i up a poli ical deba e abou some social p oblem; o
example, he AIDS p e en ion p og ams p omo ed by he
local adminis a ions we e no e y e ec i e o his pu pose,
bu hey we e good o se ing o a social deba e abou
young people’s new ways o expe iencing sexuali y. This
has been called a symbolic o poli ical use (Le i on &
Hughes, 1981).
None o hese no ions was conce ned wi h iden i ying
he u iliza ion wi hin a concep ion based on he e y p ocess
o e alua ion, which is he cen al axis o he de elopmen al
e alua ion p oposed by Pa on (1994). This e alua ion can
be conside ed a p ocess o cons an dialogue be ween
e alua o s and he pa ies in ol ed, who sha e he
esponsibili y o gene a e, ansmi , and use he in o ma ion.
E e y hing p oduces lea ning in indi iduals du ing he
e alua ion p ocess, ha is, a p ocess o cons an eedback
and change, whe e he e alua o wo ks wi h a eam o
pe sons who wan o lea n how o imp o e wha hey a e
doing and o lea n om hei ac ions.
Pa on (1997) was also in e es ed in he in en ional na u e
o he knowledge gene a ed by he e alua ion, so his
p oposal includes he iden i ica ion and di e en ia ion o
he p ima y use s om he audience in gene al, as hese
elemen s may p e e di e en usages o he e alua ion. Fo
example, a poli ician, as p ima y use , may be in e es ed in
he esul s o he e alua ion o jus i y his s ance ega ding
some social p oblem, whe eas he ci izens may be in e es ed
in he use o he esul s o he e alua ion o imp o e he
policy o dis ibu ion o goods.
All he abo e indica es he mul idi ec ional na u e o he
usage o he esul s o e alua ion, which has led Ki kha
(2000) o p opose ha i may be p e e able o alk abou
in luence ins ead o use, so as o unde line he unin en ional
and nonins umen al e ec s o he e alua ion. Fo his au ho ,
i is essen ial o de e mine he sou ces o in luence o such
elemen s o he e alua ion ha comp ise he ac i e agen s
o change and ha may be associa ed bo h wi h he p ocesses
and he esul s o he e alua ion, and also he empo al
dimension, o momen in which he in luence occu s, because
i s e ec s will a y depending on whe he i occu s du ing
o a he end o he e alua ion, o a some in e al a e he
comple ion o he e alua ion.
In sho , i is ob ious ha esea ch o he usage has aken
a new di ec ion in he las yea s, going om being a mino
issue o becoming a undamen al aspec in esea ch. This is
no su p ising i one akes in o accoun ha e alua i e wo k
should mee speci ic equi emen s bo h o he poli ical and
he scien i ic con ex , equi emen s which a e cu en ly seen
as in e wo en, hus sha e ing he a ional ou look o he
u iliza ion o he esul s o social expe imen a ion, which
had been he p edominan iewpoin since he decade o he
six ies (Campbell, 1988).
An In eg a i e App oach
A e e iewing he componen s o he e alua ion, we
p opose an o ganizing guideline in which he a ious ways
o pe o ming an e alua ion can be in eg a ed. In ou
app oach, he di e en combina ions o hese elemen s a e
implici in he goals o he e alua ion, and hese
combina ions gene a e di e en ways o add ess he
dimensions o e alua ion, as shown in Table 1.
Acco ding o his guideline, he e alua o could adop
one o se e al o he ollowing combina ions, depending on
he goals o he e alua ion:
1. Planning in e en ions.
2. Lea ning and con inuous imp o emen .
3. P og amming policies.
4. T ans o ma ion o socie y.
Planning In e en ions
E alua ions wi hin he amewo k o his goal a e mean
o ob ain in o ma ion abou he esul s achie ed by he
de elopmen o a p og am, an ac i i y, e c. An example o
his ype o e alua ions is he planned a ia ion e alua ion,
in which di e en a ie ies o a p og am a e compa ed o
de e mine hei di e en ial e icacy (see, o example, Yeh,
2000).
The goals o hese e alua ions de ine a mic o pe spec i e
and a linea change s a egy; ha is, he e alua ion ocuses
on he e aluand, wi h he aim o ob aining in o ma ion abou
he e icacy, e iciency, o e ec i eness on which o base a
decision abou u u e planning. This means elimina ing
ac ions ha ha e p o en o be ine icien o implemen ing
ac ions ha ha e been shown o be e icien .
This ype o e alua ions, suppo ed by me hodological
igo as a quali y c i e ion, equi es he objec i i y and
independence o he e alua o , who is esponsible o he
en i e e alua ion p ocess. When accep ing he s anda ds o
objec i i y and independence, he in o ma ion p o ided by
he e alua ion is expec ed o be ele an o he selec ion
and adop ion o he bes solu ions, al hough o he
unin en ional in luences may co-exis . In his case, in addi ion
o imp o ing he e icacy o a p og am, he e alua ion may
be use ul o o he la en pu poses, such as inc easing he
isibili y o he p og am in he eyes o he communi y.
Lea ning and Con inuous Imp o emen
The pu pose o his ype o e alua ions is lea ning and
con inuous imp o emen o he in e en ion. These e alua ions
ocus on he analysis o he co espondence be ween wha
EVALUATION OF SOCIAL INTERVENTION 27
was in ended and wha was ac ually implemen ed so as o
ca y ou he necessa y adjus men s o p oduce he desi ed
change in he goal o he e alua ion. The e o e, hey adop a
mic o pe spec i e and a nonlinea s a egy.
An example o his ype o e alua ions is he abo e-
men ioned heo y-d i en e alua ion, which consis s o he
o mula ion o a heo y ha desc ibes he e aluand’s
s uc u al and ope a ional cha ac e is ics in de ail, in o de
o p o ide in o ma ion abou i s na u e, unc ioning, and
aspec s ha may be subjec o change. Fo a e iew o he
mos ecen applica ions o his model, see Donaldson
(2003).
Con a y o he p e ious pe spec i e, his ype o
e alua ion has a o ma i e na u e and he pa icipa ion o
hose esponsible o he implemen a ion and ini ia ion o
he p og am being e alua ed is essen ial. They should play
a ele an ole in he speci ica ion o he e alua ion issues,
collec ing in o ma ion, in e p e ing he esul s, e c. This
pa icipa ion has ob ious e ec s on he suppo o he
p og am and on i s e alua ion, di ec ly a ec ing decision
making and p oblem sol ing.
The s ance ha he e alua o should adop abou he
alues o he g oups in ol ed can a y om dis an o nea .
In he i s case, he e alua o jus compiles, desc ibes, and
examines hese alues neu ally. In he second case, he
e alua o ies o unde s and he meaning o he p og am
o he pe sons in ol ed, because his meaning e eals which
aspec s o he p og am should be changed o imp o e i s
unc ioning.
Independen ly o he e alua o ’s s andpoin , he o she
wo ks in collabo a ion wi h he g oups in ol ed, mode a ing
and cla i ying he discussion, acili a ing decisions on he
basis o he da a ob ained.
Las ly, om his pe spec i e, he me hodological choices
will depend on he na u e o he in e en ion and on he
deg ee o speci ica ion and o maliza ion o he in e en ion.
Table 1
Guideline o he E alua ion o Social In e en ion P og ams
Goal o he E alua ion
Dimensions
E aluand
E alua ion c i e ia
Le el o pa icipa ion
E alua o ’s s andpoin
E alua o ’s esponsibili y
E alua o ’s ole
Scien i ic assump ions
Me hods o conside a ions o
he design o he e alua ion
In luence o he e alua ion
Planning in e en ions (mic o
pe spec i e, linea change)
P og ams, in e en ions o ac ions
pe o med
E icacy, e iciency o
e ec i eness
The audience is in o med o
consul ed
Independence, objec i i y
Maximum
Scien i ic
S anda ds ha accoun o
causali y
Expe imen al, quasi-expe imen al,
o co ela ional
Focused on e alua ion esul s.
P omo ed a he end o he
e alua ion.
Lea ning and con inuous
imp o emen (mic o
pe spec i e, nonlinea change)
De elopmen o he p og am,
ac ion, o in e en ion
Quali y and implemen a ion
Pa icipa ion in pa o o in
he en i e p ocess
Examines he alues
Sha ed wi h he in ol ed
g oups
Mode a o
Emphasis on he speci ica ion
o he con ex
Eme ging designs, mixed
designs, case s udies
Focused on e alua ion p ocess.
P omo ed om he s a .
P og amming policies (mac o
pe spec i e, linea change)
Se ies o in e en ions wi h
common goals
Impac
The audience is in o med o
consul ed
Independence, objec i i y
Maximum
Resolu i e
Sea ching o gene alizable
conclusions ega ding he
in luence o he con ex
Me a-analysis. Syn hesis.
Good p ac ice
Focused on e alua ion
esul s. P omo ed a he end
o he e alua ion.
T ans o ma ion o socie y
(mac o pe spec i e,
nonlinea change)
Ac ions ha lead o
disc imina ion and inequali y
Social jus ice and inclusion
Ac i e pa icipa ion.
De ense o alues
Minimal
P omo e
He meneu ical cycle o
in e p e a ion
Dialogue.
C i ical sel - e lec ion
Comp ises he inal goal o
he e alua ion
Thus, o example, when p oblems a e no well de ined,
people disag ee abou which aspec s should ha e p io i y,
he goals a e di use, o he ac i i ies a e no implemen ed
in hei o ali y, hen he e alua o should s a ou wi h
na u alis ic assump ions so as o disco e and unde s and
he phenomena being s udied in hei con ex . Acco ding o
his iewpoin , i is no necessa y o begin wi h a p io
design, bu ins ead a design “eme ges” and akes shape as
he e alua ion ad ances; quan i a i e measu es can
subsequen ly be inco po a ed o inc ease he c edibili y o
he indings (see o example Kala a & Illback, 1998). On
he con a y, when he p og am is highly speci ic and
o malized, one should choose quan i a i e me hods om
he s a .
P og amming Policies
In his g oup a e he e alua ions whose pu pose is o
o mula e gene al in e en ion p inciples o help imp o e
social policies and compile he bes p ac ices o lessons
lea ned. In his case, he e alua ions go beyond he limi s
o a conc e e p og am o ac ion, adop ing a mac o
pe spec i e aimed a poli ical, social, o ganiza ional, and
psychological ac o s ha ha e acili a ed o hinde ed he
achie emen o he bes esul s (linea s a egy).
The e a e wo op ions he e: he i s consis s o
pe o ming a me a-analysis o he indi idual wo ks and
in eg a ing he indings ob ained by each o he pa icula
e alua ions and e ealing e ec s ha would o he wise no
be obse able (see, o example, Lipsey, 1997; Lipsey &
Wilson, 1993). The second op ion consis s o compa ing a
se ies o in e en ions wi h common aims and cha ac e is ics
bu aking place in di e en con ex s. This ype o e alua ion
is known as clus e e alua ion (see Sande s, 1997).
The pu pose o bo h ypes o s udies is o ob ain
gene alizable indings ha lead o be e comp ehension o
he in e en ion p og ams by analyzing he di e se
ci cums ances and con ex s in which hese in e en ions
achie e he bes esul s. F om his iewpoin , he e alua o s
do no limi hei ole o ha o me e ad iso s in he
decision-making p ocess o de elop new ini ia i es, policies,
o implemen a ion s a egies, bu ins ead hey also make
sugges ions o sol e he p oblems.
T ans o ma ion o Socie y
As in he p e ious case, his ype o e alua ions a e
wi hin he socio-poli ical con ex , bu con e sely, hey
exp ess misgi ings abou he main aspec s o he p og ams
in o de o p omo e subs an ial changes in alues and
poli ical p io i ies (mac o pe spec i e). Thus, hey gene a e
a pe manen deba e abou he p ocesses ha e eal he
weaknesses o he cu en social sys em (nonlinea change
s a egy) so as o sol e he mos u gen social p oblems:
po e y, isola ion, delinquency, e c. Fo example, he main
pu pose o he eminis e alua ion is o make explici he
elemen s ha explain gende inequali y (Sielbeck-Bowen,
B isola a, Seiga , Tischle , & Whi mo e, 2002).
Fo his pu pose, he e alua o adop s an ac i is s ance
ha de ends he in e es s o he g oups in ol ed, in pa icula
o hose who lack powe and igh s, and p omo es he ac i e
pa icipa ion o hese g oups o ans o m he exis ing
s uc u es. Con e sely, he e alua ion is based on deba e,
dialogue, and e lec ion wi hin he he meneu ical cycle o
obse a ions and in e p e a ions ha lead o he unde s anding
and comp ehension o eali y.
Las ly, gi en he c i ical examina ion o he alues and
assump ions o he p og ams and he need o make he
cu en inequali ies explici , he e alua ion plays an ac i e
ole in poli ical ac i i y, because he goal is o he e alua ion
o be linked o social ans o ma ion.
Summing up, he pu pose o his guideline is o he
e alua ion o be comp ehensi e and ealis ic; ha is, o
accep ha he e alua ion mee s he demands and needs o
a ce ain social con ex and he goals o sol ing social
p oblems, using he e alua ion heo y ha is mos app op ia e
o one’s pu poses. This pe spec i e, a he han b eaking
up he ield o e alua ion, should, o a g ea ex en , con ibu e
o he p oduc i e p og ess and plu ali y o in e es s,
pe spec i es, and alues o socie y.
Re e ences
Ál a o, J.L. (1995). Psicología social: pe spec i as eó icas y
me odológicas. Mad id: Siglo XXI.
Al i a, F. (1983). Pe spec i a cuan i a i a-cuali a i a en la
me odología sociológica. Re is a española de in es igaciones
sociológicas, 22, 53-75.
Bickman, L. (2000). Summing up p og am heo y. New Di ec ions
o E alua ion, 87, 103-111.
Bi ckmaye , J.D., & Weiss, C.H. (2000). Theo y-based e alua ion
in p ac ice. E alua ion e iew, 24, 407-431.
Campbell, D.T. (1988). Me hodology and epis emology o social
science. Selec ed pape s. Chicago: Uni e si y o Chicago.
Chelismky, E., & Shadish, W.R. (Eds.) (1997). E alua ion o he
21s cen u y. A handbook. Thousand Oaks, CA: Sage.
Chen, H.T. (1990). Theo y-d i en e alua ions. Newbu y Pa k, CA:
Sage.
Cook, T.D. (1997). Lessons lea ned in e alua ion o e he pas 25
yea s. In E. Chelimsky & W. Shadish (Eds.), E alua ion o
he 21s cen u y: A handbook (pp. 30-52). Thousand Oaks, CA:
Sage.
Cook, T.D. (2000). The alse choice be ween heo y-based
e alua ion and expe imen a ion. New Di ec ions o E alua ion,
87, 27-34.
C onbach, L.J. (1982). Designing e alua ions o educa ional and
social p og ams. San F ancisco: Jossey-Bass.
Donaldson, S.I. (2003). Theo y-d i en p og am e alua ion in he
new millennium. In S.I. Donaldson & E.M. Sc i en (Eds.),
HERRERA, LEÓN, AND MEDINA
28
EVALUATION OF SOCIAL INTERVENTION 29
E alua ing social p og ams and p oblems: Visions o he new
millennium (pp. 109-141). Mahwah, NJ: E lbaum.
Fe e man, D.M. (2001). Founda ions o empowe men e alua ion.
Thousand Oaks, CA: Sage.
Guba, E.G., & Lincoln, Y.S. (1989). The ou h-gene a ion
e alua ion. Newbu y Pa k, CA: Sage.
House, E.R. (1980). E alua ing wi h alidi y. Be e ly Hills, CA:
Sage.
House, E.R. (2001). Responsi e e alua ions (and i s in luence on
delibe a i e democ a ic e alua ion). New Di ec ions o
E alua ion, 92, 23-30.
House, E.R., & Howe, K.R. (2000). Delibe a i e democ a ic
e alua ion. New Di ec ions o E alua ion, 85, 3-12.
Julnes, G., & Ma k, M.M. (1998). E alua ion as sensemaking:
Knowledge cons uc ion in a ealis wo ld. New Di ec ions o
E alua ion, 78, 33-52.
Kala a , J., & Illback, R. (1998). A quali a i e e alua ion o school-
based amily esou ce and you h se ice cen e s. Ame ican
Jou nal o Communi y Psychology, 26, 573-604.
Ki kha , K.E. (2000). Reconcep ualizing e alua ion use: An
in eg a ed heo y o in luence. New Di ec ions o E alua ion,
88, 5-23.
Le i on, L.C., & Hughes, E.F. (1981). Resea ch on he u iliza ion
o e alua ions: A e iew and syn hesis. E alua ion Re iew, 5,
525-548.
Lipsey, M.W. (1997). Wha can you build wi h housands o b icks?
Musings on he accumula ion o knowledge in p og am
e alua ion. New Di ec ions o E alua ion, 76, 7-23.
Lipsey, M.W., & Co d ay. D.S. (2000). E alua ion me hods o
social in e en ion. Annual e iew o Psychology, 51, 345-375.
Lipsey, M.W., & Wilson, D.B. (1993). The e icacy o
psychological, educa ional and beha io al ea men :
Con i ma ion om me a-analysis. Ame ican Psychologis , 48,
1181-1209.
Ma k, M.M. (2001). E alua ion’s u u e: Fu o , u ile o e ile?
Ame ican Jou nal o E alua ion, 22, 457-479.
Ma k, M.M., Felle , I., & Bu on, S. (1997). In eg a ing quali a i e
me hods in a p edominan ly quan i a i e e alua ion. A case
s udy and some e lec ions. New Di ec ions o E alua ion,
74, 47-59.
Me ens, D.M. (2003). The inclusi e iew o e alua ion: Visions
o he new millennium. In S. I. Donaldson & E. M. Sc i en
(Eds.), E alua ing social p og ams and p oblems: Visions o
he new millennium (pp. 91-107). Mahwah, NJ: E lbaum.
Pa on, M. Q. (1994). De elopmen al e alua ion. E alua ion
P ac ice, 15, 311-319.
Pa on, M.Q. (1997). U iliza ion- ocused e alua ion: The new
cen u y ex (3 d Ed.). Thousand Oaks, CA: Sage.
Pa on, M.Q. (2001). E alua ion, knowledge managemen , bes
p ac ices, and high quali y lessons lea ned. Ame ican Jou nal
o E alua ion, 22, 329- 336.
Rebolloso, E., & Rebolloso, J.R. (1998). Signi icado y desa ollo
ac ual de la e aluación de p og amas. In E. Rebolloso (Ed.),
E aluación de p og amas: ámbi os de aplicación (pp. 9-29).
Ba celona: Tex os Uni e si a ios “Sain Jo di.”
Reicha d , C.S., & Cook, T.D. (1979). Beyond quali a i e e sus
quan i a i e me hods. In T.D. Cook & C.S. Reicha d (Eds.),
Quali a i e and quan i a i e me hods in e alua ion esea ch
(pp. 7-32). Be e ly Hills, CA: Sage.
Rossi, P.H., F eeman, H.E., & Lipsey, M.W. (1999). E alua ion.
A sys ema ic app oach (6 h ed.). Thousand Oaks, CA: Sage.
Sande s, J.R. (1997). Clus e E alua ion. In E. Chelimsky & W.
Shadish (Eds.), E alua ion o he 21s cen u y: A handbook
(pp. 396-404). Thousand Oaks, CA: Sage.
Schwand , T.A. (1997). The landscape o alues in e alua ion:
Cha ed e ain and unexplo ed e i o y. New Di ec ions o
E alua ion, 76, 25-37.
Sc i en, M. (1980). The logic o e alua ion. In e ness, CA:
Edgep ess.
Sc i en, M. (1994). The inal syn hesis. E alua ion P ac ice, 15,
367-382.
Shadish, W.R. (1987). P og am mic o- and mac o heo ies: A guide
o social change. New Di ec ions o P og am E alua ion, 33,
93-109.
Shadish, W.R., Cook, T.D., & Le i on, L.C. (1991). Founda ions
o p og am e alua ion: Theo ies o p ac ice. Newbu y Pa k,
CA: Sage.
Sielbeck-Bowen, K.A., B isola a, S., Seiga , D., Tischle , C., &
Whi mo e, E. (2002). Explo ing eminis e alua ion: The g ound
om which we ise. New Di ec ions o E alua ion, 96, 3-8.
S ake, R.E. (1983). P og am e alua ion, pa icula ly esponsi e
e alua ion. In G.F. Madaus, M.S. Sc i en, & D.L. S u lebeam
(Eds.), E alua ion models: Viewpoin s on educa ional and
human se ices e alua ion (pp. 287-310). Bos on: Kluwe -
Nijho .
S ake, R.E., & Migo sky, C. (1997). The e ol ing syn heses o
p og am alue. E alua ion P ac ice, 18, 89-103.
S u lebeam, D.L. (2001). E alua ion models. New Di ec ions o
E alua ion, 89, 7-98.
Vedung, E. (1993). Modelos de e aluación. Re is a de se icios
sociales y polí ica social, 30, 39-68.
Veney, J.E., & Kaluzny, A.D. (1991). E alua ion and decision
making o heal h se ices p og am (2nd Ed.). Ann A bo , MI:
Heal h Adminis a ion P ess.
Weiss, C.H. (1997). How can heo y-based e alua ion make a
g ea e headway? E alua ion Re iew, 21, 501-524.
Weiss, C.H. (1998). E alua ion (2nd Ed.). Englewood Cli s, NJ:
P en ice Hall.
Weiss, C.H., & Bucu alas, M.J. (1981). T u h es and u ili y es s:
Decision-make s ame o e e ence o social science esea ch.
E alua ion S udies Re iew Annual, 6, 695-706.
Wholey, J.S. (1983). E alua ion and e ec i e public managemen .
Bos on: Li le B own.
Yeh, S.S. (2000). Building he knowledge base o imp o ing
educa ional and social p og ams h ough planned a ia ion
e alua ions. Ame ican Jou nal o E alua ion, 21, 27-40.
Recei ed: May 26, 2004
Re ision ecei ed: No embe 11, 2004
Accep ed: No embe 30, 2004