A pragmatic guideline for evaluation of social intervention

Conside ing he many heo e ical and me hodological iewpoin s in he ield o e alua ion, a guideline was es ablished o acili a e he e alua ion o social in e en ion p og ams. Fo his pu pose, he goals o he e alua ion we e aken in o accoun : (a) planning in e en ions, (b) lea ning and con inuous imp o emen o in e en ions, (c) p og amming policies, and (d) ans o ma ion o socie y. These goals will de e mine he pe spec i e o he analysis selec ed ( ocusing on he e aluand o he con ex ) and he s a egies o change employed ( ocusing on p ocesses o esul s). The elemen s ha , acco ding o Shadish, Cook, and Le in on (1991), cons i u e he heo y o e alua ion (e aluand, alue, building o knowledge, and uses) ha e also been conside ed. The analysis o all hese componen s led o he elabo a ion o a guideline o o ien he p ac ice o e alua ion om a p agma ic pe spec i e, in acco dance wi h he demands and needs o a ce ain social con ex . Keywo ds: p og am e alua ion, social in e en ion, e alua ion heo y, e alua ion p ac ice Conside ando la mul iplicidad de pe spec i as eó icas y me odológicas exis en es en el campo de la e aluación, se ha elabo ado una pau a que pe mi a conduci la e aluación de p og amas de in e ención social. Pa a ello se han enido en cuen a los p opósi os u obje i os que puede pe segui la e aluación: (a) plani icación de in e enciones, (b) ap endizaje y mejo a con inua de las in e enciones, (c) p og amación de polí icas, y (d) ans o mación de la sociedad. Es os obje i os de e mina án la pe spec i a de análisis adop ada (cen ada en el e aluando o en su con ex o) y las es a egias de cambio u ilizadas (o ien adas a los p ocesos o a los esul ados). Se han conside ado ambién los elemen os que pa a Shadish, Cook y Le in on (1991) cons i uyen la eo ía de la e aluación (e aluando, alo , cons ucción del conocimien o y usos). El análisis de odos es os componen es ha pe mi ido elabo a una guía que o ien e la p ác ica de la e aluación desde una pe spec i a p agmá ica aco de a las demandas y necesidades que se plan een en un con ex o social de e minado. Palab as cla e: e aluación de p og amas, in e ención social, eo ía de la e aluación, p ác ica de la e aluación A P agma ic Guideline o E alua ion o Social In e en ion Isabel Ma ía He e a Sánchez, José Ma ía León Rubio, and Sil ia Medina Anzano Uni e si y o Se ille The Spanish Jou nal o Psychology Copy igh 2005 by The Spanish Jou nal o Psychology 2005, Vol. 8, No. 1, 21-29 1138-7416 Co espondence should be add essed o: Isabel Ma ía He e a Sánchez. Depa amen o de Psicología Social. Uni e sidad de Se illa. Camilo José Cela s/n. 41018 Se illa (Spain). Phone: 95455988. E-mail: [email p o ec ed] T ansla ion: Vi ginia Na ascúes Howa d. 21 The pu pose o his wo k is o p o ide a guideline o o ien e alua o s abou he bes way o pe o m an e alua ion in he a ea o social in e en ion, depending on he goals, as hese de e mine how he main e alua ion elemen s a e se up. Howe e , be o e going in o de ails and o make ou p oposal mo e comp ehensible, we shall de ine social in e en ion as a se ies o planned s a egies o achie e changes in he social eali y in a ce ain popula ion, so ha his popula ion will each highe le els o de elopmen and well-being. Thus, he e alua ion o any social in e en ion p og am in ol es acqui ing knowledge abou he alue o he p og am and ans e ing such a judgmen o sol ing social p oblems. The a ionale o ou p oposal is based on e alua o s’ di icul y in choosing he e alua ion p ocedu e ha bes ma ches hei demands and needs, gi en he quan i y o a ailable s a egies and hei di e si y (e.g., Chelimsky & Shadish, 1997; S u lebaum, 2001). A esea che ’s i s ask is o ansla e eme ging demands and needs in o goals. These goals can be g ouped in o one o he ollowing ca ego ies: (a) planning in e en ions, (b) lea ning and con inuous imp o emen o in e en ions, (c) p og amming policies, and (d) ans o ma ion o socie y. Inclusion in any ca ego y will de e mine he analysis ou look (mic o o mac o) assumed by he esea che and he change s a egy adop ed (linea o nonlinea ). The “mic o” pe spec i e ocuses on s uc u al ( esou ces, pe sonnel, e c.) and ope a ional (ac i i ies, s a egies, e c.) cha ac e is ics o he in e en ion p og am ha is he objec o e alua ion, in o de o de e mine which aspec s a e imp o able. The “mac o” pe spec i e add esses he poli ical, economic, o ganiza ional, o psychosocial aspec s o a p og am (see Shadish, 1987). Change s a egies e e o he ime pe iod in which he e alua ion akes place (a he end o du ing he en i e p ocess) and o he ype o e alua ion pe o med (p ocess- o ou come- ocused). Acco ding o Veney and Kaluzny (1991), linea s a egies lead o punc ual e alua ions pe o med o achie e conc e e goals ha a e usually aimed a p esen ing accoun s and inc easing he e icacy o impac o social p og ams. They a e he e o e ypically pe o med a he end o he in e en ions. Nonlinea s a egies conside e alua ion and in e en ion o be pa o he same p ocess ha goes om he in e en ion o he e alua ion, and back again, so ha a con inued e alua ion o he implici p ocesses o he p og am is pe o med in o de o in oduce he pe inen changes o op imize he in e en ion. Ob iously, e alua ions may ha e a ious goals so ha in one e alua ion, se e al di e en pe spec i es o analysis and change s a egies may be adop ed. Fo example, an e alua ion may p opose o: (a) e i y whe he he p og ammed ac i i ies a e being adequa ely implemen ed o achie e con inuous imp o emen (mic o pe spec i e, nonlinea change); (b) de e mine he e icacy o he p og am so as o decide whe he o con inue i , modi y i , o elimina e i (mic o pe spec i e, linea change); (c) e alua e he di e en ial e ec o di e se p og ams wi h simila cha ac e is ics on di e en se ings (mac o pe spec i e, linea change); and (d) assess he ex en o which a ious p og ams ha a e being ca ied ou p omo e democ a ic alues such as solida i y, ole ance, and espec o human igh s (mac o pe spec i e, nonlinea change). In sho , he goals o an e alua ion de e mine he analysis pe spec i e adop ed and he way he e aluand is expec ed o change. Howe e , i s ill emains o be seen how o pe o m he e alua ion o achie e he p oposed goals. This leads o he examina ion o he elemen s ha make up he e alua ion, as hey will de e mine he decisions o be made. Fo his pu pose, we chose he e alua ion heo y o mula ed by Shadish, Cook, and Le i on (1991), which we hink is one o he mos solid and comple e heo e ical o mula ions. I compiles he con ibu ions o he chie e alua ion heo e icians o he imes—Sc i en, Campbell, Weiss, Wholey, S ake, C onbach, and Rossi—conside ing he ollowing elemen s: (a) he e aluand’s cha ac e is ics; (b) he e alua ion c i e ia, he pa icipa ion le el o he g oups in ol ed in he e alua ion, he s andpoin , esponsibili y and ole assumed by he e alua o ; (c) he e alua o ’s scien i ic assump ions and he me hods used o ob ain in o ma ion; and (d) he use o in luence o he in o ma ion p o ided by he e alua ion. The examina ion o each one o hese componen s will allow us o decide wha o do and how o ac ually conduc he e alua ion depending on he goals pu sued. We ha e adop ed a p agma ic s andpoin o he analysis, as we belie e ha all he op ions a e legi ima e as long as hey sa is y he needs encoun e ed by he e alua o in he di e en se ings in which he e alua ion akes place. The E aluand’s Cha ac e is ics The sys ema iza ion o he e alua ion p ocess in ol es conside ing, i s , he objec o he e alua ion—o e aluand—, which, in ou case, a e he social in e en ion p og ams. We mus he e o e de e mine he main assump ions and mechanisms unde lying he in e en ion: he componen s (assigned esou ces, in e en ion s a egies and ac ics, esul s), he expec ed causal ela ions among hem, and he con ex ual ac o s ha accompany hei implemen a ion ( alues, social no ms, ela ion wi h o he sociopoli ical ac ions, p essu e om a ious s akeholde s, e c.). The so-called heo y-d i en e alua ion is a model ha has ocused mos di ec ly on knowledge o he e aluand. This mo emen eme ged a he end o he eigh ies as an al e na i e o he expe imen al models o black box e alua ion (Chen, 1990), because o i s p oblema ical applica ion o he e alua ion o social and communi y in e en ions. This heo y is based on he no ion ha he design o an e alua ion should es on he logical model HERRERA, LEÓN, AND MEDINA 22 ha di ec s he p og am o in e en ion i is mean o e alua e. This means aking in o conside a ion he p ocesses in ol ed in o mula ing he implemen a ion o he in e en ion, as well as he esul s expec ed o be achie ed by he in e en ion. The app aisal o he p ocesses chie ly in ol es de e mining whe he he heo y unde lying he p og am allows one o p opose he desi ed goals and whe he he implemen a ion o he p og am has been ca ied ou acco ding o plan. The wo issues ha he no ma i e heo y (Chen, 1990), he implemen a ion heo y (Weiss, 1998), and he p ocess heo y (Rossi, F eeman, & Lipsey, 1999) a e expec ed o cla i y a e he heo y’s su iciency and he p og am’s adap a ion o i s concep ion In he case o he e alua ion o he esul s, he emphasis is on iden i ying he causal mechanisms ha media e be ween he componen s o a p og am and he esul s. The issues ha conce n he causa i e heo y (Chen, 1990), he p og am heo y (Weiss, 1998), and he impac heo y (Rossi e al., 1999) a e he e icacy, e iciency, and e ec i eness o he p og am. No doub , his is he bes ype o e alua ion o in es iga e he concep ion o a p og am and o ga he exhaus i e knowledge o he e aluand. Howe e , i also p esen s a se ies o limi a ions, among hem: (a) lack o p ecision in he elabo a ion o he logical models ha guide he p og am, among o he easons, because he same p og am may be based on se e al implici heo ies; o he e alua o , who may no be in he bes si ua ion o conside all he con ex ual ac o s o he p og am, may neglec ele an a iables when concei ing he p og am (Bickman, 2000; Bi ckmaye & Weiss, 2000); (b) he models esul ing om he esea ch p ocess pe o med by he e alua o a e so complex ha i may be impossible o ansla e hem in o a da a analysis design (Weiss, 1997); and (c) knowledge o he concep ion o a p og am does no exemp esea che s om using expe imen al designs o con i m i (Cook, 2000). Despi e all hese limi a ions, we belie e ha he elabo a ion o logical models is usually one o he i s asks o be add essed by he e alua o . Howe e , he p oblem consis s o de e mining which elemen s o he p og am should be ope a ionalized wi hin he logical model and hei impo ance in he se ies o ac i i ies ha make up an e alua ion. The highe o lesse deg ee o impo ance o he logical model will depend on he change s a egy adop ed: 1. Ou come- ocused e alua ion. This may no equi e a logical model, as i can s udy he e ec s o an in e en ion on a ce ain popula ion, wi hou he need o app aising he unde lying mechanisms o he componen s o he p og am and he esul s achie ed. 2. P ocess- ocused e alua ion. The e alua ion ocuses on he ope a ionaliza ion o he heo y; he e alua o should de e mine whe he he p og am begins wi h a heo e ical analysis o he p oblem, whe he i akes in o accoun how he a ious g oups in ol ed (manage s, p og amme s, echnicians, use s, e c.) concei e he p oblem, o whe he he heo y is compa ible wi h he sys em o dominan alues, belie s, and ideologies. Fo example, an e alua ion may ecommend he need o know whe he a p og am o i s componen s allow o he e alua ion o he esul s (e alua ion o he e aluabili y). 3. P ocess- and esul s- ocused e alua ion. An example is he p oposal o Lipsey and Co d ay (2000) o he e alua ion o he esul s, which in eg a es an expe imen al o quasi-expe imen al design wi h heo y-d i en e alua ion, p o iding mo e p ecise in o ma ion abou he causal mechanisms o he changes p oduced by he p og am. Tha is: (a) obse ed a iabili y a ibu ed o ac o s such as he implemen a ion o he p og am (o ganiza ion o ac i i ies de eloped and consis en wi h wha was planned), he pa icipan s’ cha ac e is ics, he ype o con ac wi h se ice p o ide s, e c.; and (b) he cause- e ec sequence be ween p og am componen s and hei in e ela ions wi h p og am esul s. E alua ion C i e ia, Le el o Pa icipa ion o he G oups In ol ed, and he E alua o ’s Role Judgmen s a e an omnip esen ac i i y in any e alua ion ask. Judgmen is doub less implici in e alua ion, a leas , o a compa a i e na u e. This means one mus explici ly exp ess he logic used o es ablish he c i e ia ha lead o he o mula ion o he conclusions o he e alua ion. This emphasis on assessmen and judgmen o he e aluand is wha dis inguishes e alua ion om esea ch. This led Sc i en (1980) o es ablish objec i e e alua ion c i e ia, de eloping a me a- heo y ha he denomina ed he Logic o E alua ion. Acco ding o his au ho , he in o ma ion p o ided by he e alua ion is ela ed o ce ain judgmen s in ou phases: (a) es ablishmen o he c i e ia o me i ha de e mine how he objec o he e alua ion will be labeled and judged as good, use ul, o quali y, e c.; (b) elabo a ion o s anda ds o no ms o unc ioning ha speci y le els o deg ees o me i ; (c) collec ing da a abou he unc ioning o he objec and compa ing i wi h p e iously es ablished unc ioning s anda ds; and (d) syn hesis o in eg a ion o he da a wi hin a judgmen o me i o alue. This logic o e alua ion is conside ed a ool ha is uni e sally applicable o he p ac ice o e alua ion (S ake & Migo sky, 1997). A gumen s a ise when choosing he mos pe inen c i e ia o de e mine he alue o me i o he e aluand, deciding he app op ia eness o he s anda ds o no ms o unc ioning, and deciding how o measu e he le el o unc ioning. Howe e , he bigges p oblems c op up when a emp ing o o mula e he global judgmen om he syn hesis o he da a collec ed. Thus, whe eas o some au ho s, alid and objec i e in e ences abou alues can be ob ained om he da a (Sc i en, 1994), o o he s, his is EVALUATION OF SOCIAL INTERVENTION 23 impossible because hey conside he da a o be a p ocess o social cons uc ion and he e o e, subjec o in e p e a ion by he people in ol ed (Guba & Lincoln, 1989; S ake, 1983; S ake & Migo sky, 1997). This leads us o he conside a ion o whe he he e alua o should se e some p e-es ablished absolu e alues, o jus desc ibe he alues o ce ain s akeholde s. In his sense, om a p esc ip i e app oach, we p opose he p imacy o uni e sal alues ha ci cumsc ibe he e alua ion. The au ho s who de end his op ion, o example, House (1980), begin wi h he p emise ha when e alua o s p o ide in o ma ion abou ce ain ac ions aimed a sol ing social p oblems (po e y, acism, iolence, e c.), hey a e pa icipa ing in a mo al ac ha should con ibu e o jus ice in socie y in gene al. Va ious di icul ies a ise i we decide o conside alues as absolu e. Fi s , alues a e no limi ed o a heo y o philosophical p oposal, so i is impossible o jus i y one heo y o e ano he based on alues. Second, in e alua ion, he common p ac ice is o ha e o deal wi h di e en alues and in e es s ha co espond o di e en g oups o people in ol ed in he ac ions being e alua ed (Guba & Lincoln, 1989; Rebolloso & Rebolloso, 1998; Schwand , 1997). In his sense, he s andpoin o he majo i y conside s i mo e p ac ical o de end a desc ip i e app oach: E alua ion is limi ed o desc ibing o no ing his plu ali y o in e es s, es ablishing he c i e ia o alue on he basis o wha ce ain s akeholde s conside o pe cei e o be impo an o mo e jus . P ecisely, he p esence o a ious s akeholde s ha pa icipa e in he e alua ion p ocess is one o he main p oblems, because his plu ali y implies he equen con e gence o con lic ing alues. This si ua ion o ces one o make decisions abou he need o conside conjoin ly he in e es s o all he people who pa icipa e in he ac ion, o o choose he mos ele an among hem. The e a e se e al iewpoin s on his issue. So, some e alua ion models, such as he so-called s akeholde e alua ion, de end he need o “holis ically” conside he conce ns and needs all he a ec ed g oups (Vedung, 1993). Con e sely, o he s gi e p io i y o he alues o ce ain g oups, such as, o example, p og am and se ice manage s (Wholey, 1983), people who lack powe and who a e no ep esen ed impa ially in democ a ic decision making (House & Howe, 2000), o mino i y g oups and hose who ac in hei name (Me ens, 2003). Along wi h he conside a ion o he a ious g oups ha a e in e es ed in he e alua i e p ocess as a sou ce o in o ma ion, he di e en le els o in ol emen o hese g oups in he e alua ion ask and he e alua o ’s ole a e also pa icula ly ele an . The e a e also se e al choices he e. Thus, in he esponsi e e alua ion model, al hough i con empla es he pa icipa ion o he g oups in ol ed in he e alua ion, he e alua o is esponsible o and con ols he e alua ion (House, 2001). Con e sely, in he cons uc i is app oach o Guba and Lincoln (1989), bo h e alua o s and he people in ol ed sha e he esponsibili y in a nego ia ion iewpoin o he e alua i e p ocess. Las ly, he well-known empowe men e alua ion (Fe e man, 2001) places he s akeholde s in he cen e o he e alua ion p ocess, looking o he sel -de e mina ion o he people who pa icipa e in he p og am; ha is, hei capaci y o de e mine hei own li e cou ses. This is a model o pa icipa ion and ans o ma ion in which he e alua o assumes he ole o p omo e o he lea ning and change p ocesses. The inclusion o he g oups in ol ed in he e alua ion p ocess and he ype o ela ion hey ha e wi h he e alua o ha e eopened a deba e among heo e icians abou he s andpoin ha he e alua o should main ain du ing he e alua ion p ocess. I is a deba e o objec i i y e sus commi men . Some au ho s s a e ha e alua ion should be neu al, in his case, he e alua o s examine sys ema ically and empi ically he alues ha a e ele an o he social policies and p og ams and, i hey ind disc epancy, hey y o a i e a a consensus be ween he di e en posi ions o he g oups (Ma k, 2001). Con e sely, o he au ho s de end a s andpoin ha is mo e commi ed o a ce ain ype o e alua ion, ha which ad oca es he speci ic in e es s o ce ain social g oups, in pa icula , hose lacking powe and igh s (Me ens, 2003). In sho , hese p oposals unde sco e he di icul y o assigning p io i y o he alues ha suppo he e alua ion, o choosing and in ol ing he s akeholde s, and o de e mining he e alua o ’s habi ual ole. Once again, i is no ed ha ma ching he e alua ion o he goals can help sol e his p oblem. Thus, i he aim is o de e mine he e icacy o a p og am (mic o pe spec i e, linea change), he alues will al eady be speci ied because hey a e implici in he in e en ion goals. The e o e, i is no necessa y o he g oups in ol ed o pa icipa e in his aspec . Simila ly, he e alua o ’s ole should be neu al because he esul s o he e alua ion will depend on he causal in e ence being unbiased. Howe e , i he goal is o iden i y and de e mine which p og am ou comes will be ele an o he a ious g oups in ol ed when subsequen ly designing he in e en ion (mic o pe spec i e, nonlinea change), hen he alues should be ag eed on by hese g oups, which equi es hei maximum pa icipa ion in he in es iga ion p ocess. The e alua o should hen adop a ole o media o o nego ia o who collec s in o ma ion abou he di e en needs and who a emp s o each ag eemen s among he a ious c i e ia and p io i ies p oposed by he g oups in ol ed. Scien i ic P oposals and Me hods The key issue in his sec ion is how o ob ain he knowledge o judge he e aluand; whe he such knowledge should be ga he ed using quali a i e me hodology such as ha employed in idiog aphic disciplines, o whe he i should be based on he adminis a ion o quan i a i e analysis p ocedu es employed in nomo he ic disciplines. In o he HERRERA, LEÓN, AND MEDINA 24 EVALUATION OF SOCIAL INTERVENTION 25 wo ds, he issue is whe he he objec o e alua ion is o unde s and he e aluand o o o mula e gene al laws, ansla able o wha Pa on (2001) calls he e ec i e p inciples o p og amming social in e en ions. Con a y o ex eme iewpoin s ha ega d bo h me hodological app oaches as opposi es, we ad oca e a iewpoin ha does no conside hem dicho omic bu complemen a y. This ou look s a es ha he de ini ion o he objec o s udy and he goals o he in es iga ion de e mine he limi s o choice, bo h o he me hodological app oach and o he analysis echniques (see, o example, Cook, 1997). The e o e, al hough he epis emological bases o hese wo me hodological adi ions a e di e en , he e is no con adic ion in hei conjoin use in social in es iga ion (Ál a o, 1995; Al i a, 1983; Reicha d & Cook, 1979). This pe spec i e has p oduced wo speci ic p oposals in he ield o e alua ion: 1. The use o a speci ic ype o me hodology and i s co esponding analysis echniques, which does no imply he accep ance o a whole se ies o me a- heo e ical o epis emological p oposals. Fo example, he u iliza ion- ocused e alua ion (Pa on, 1997), a model di ec ed a using he e alua ion ou come, so he choice o possible me hodological al e na i es (expe imen a ion, quasi-expe imen a ion, s a is ical designs, case s udies, e c.) will depend on whe he o no he da a ob ained mee he use s’ needs. Fo ins ance, i he goal o an in e en ion p og am was he academic in eg a ion o e hnical mino i y g oups, i migh be su icien o hose in cha ge o know he equency o in e ac ions be ween s uden s om hese g oups and hose om he majo i y g oup; in his case, s a is ical designs would be he mos app op ia e. Howe e , i he manage s needed o know he na u e o such in e ac ions, hen obse a ion in si u and in- dep h in e iews would be equi ed. 2. The co-exis ence o con as ing me hodological app oaches and analysis echniques (causal and in e p e i e in es iga ion) in he same heo e ical p oposal, as occu s, o example, in he ealis ic e alua ion (Julnes & Ma k, 1998). This e alua ion model is based on eme gen ealism, which s a es ha he e is an ex e nal eali y om which obse able egula i ies may be ex ac ed, which should be conside ed wi hin he con ex ual dynamic o complex open sys ems ha a e obse ed indi ec ly. The e o e, he alue o he di e en me hods will depend on hei capaci y o e i y he di e en aspec s o he gene a i e change mechanisms associa ed wi h he social p og ams and he di e se aspec s o he con ex a ec ed by he p og am. An example is he case s udy p esen ed by Ma k, Felle , and Bu on (1997), in which he impac o a new sys em o con ac ing pe sonnel in he Adminis a ion was e alua ed. Fo his pu pose, a se ies o quali a i e me hods (obse a ions and in e iews) we e inco po a ed in he amewo k o an expe imen al design. These quali a i e me hods had se e al goals, among hem: de elopmen and subsequen e ining o he measu emen ins umen s (su eys), alida ion o he esul s ob ained in he su eys, explana ion o he quan i a i e indings, and se ing he limi s o hei gene aliza ion. Rega ding he las poin , in he in e iews wi h he di ec o s and he new employees, he au ho s cla i ied impo ance o ac o s such as p o essional ca ego y, sala y, wo k condi ions, and con ac es ic ions o de e mine whe he o no i was necessa y o implemen he new con ac ing sys em. We ad oca e he p oposal o Pa on (1997) and we coincide wi h his opinion ha he me hodology should be selec ed acco ding o he use s’ needs, which a e e lec ed in he e alua ion goals. To a oid subjec ing he e alua ion o p i a e in e es s, hese goals should be es ablished as a esul o wha he use s need o know and he e alua o conside s mos pe inen , gi en he peculia i ies o he objec o he e alua ion. Thus, a he s a o he p og am, when li le is known abou he p oblem and i s solu ions, i is mos app op ia e o disco e and desc ibe he peculia i ies o he p oblem ( o example, by case s udies). When a p og am is su icien ly implemen ed a a local le el o be able o o e an accoun o i s e icacy, he e alua ion will equi e a causal o p obabilis ic explana ion (expe imen al, quasi- expe imen al, o co ela ional designs). I he di e en ial e ec s o simila p og ams in di e en se ings should be de e mined so as o ensu e he con inua ion o some poli ical s a egy, he e alua ion should lead o gene alizable conclusions (such as he case o he syn hesis o esea ch o me a-analysis). Las ly, in o de o go beyond he impac o he p og am and ake in o accoun he alue o a ious possible al e na i es o sol e he social p oblems, he e alua ion should be o an in e p e i e na u e, using a he meneu ic model. In sho , we belie e ha e alua ion should allow one o es ablish a se ies o gene al p inciples applicable o any social in e en ion, wi hou ha ing o gi e up unde s anding he speci ic e aluand in which he e alua ion o igina ed. The U iliza ion o In luence o he In o ma ion P o ided by he E alua ion Resea ch in his ield has ocused on iden i ying how he indings o an e alua ion a ec he di e en s akeholde s. Ini ially, he implici use heo ies s a ed ha he esul s o an e alua ion should become he cen al inpu s in poli ical decision making (ins umen al use). Howe e , expe ience soon showed he e alua o s ha poli ical decisions we e no made acco ding o he esul s o he e alua ion; some imes because he esul s we e con a y o poli ical in e es s and o he s because hey we e no de ini e. HERRERA, LEÓN, AND MEDINA 26 Al hough he esul s do no lead o new ac ions o o he modi ica ion o he e alua ed ac ions, hey may be used o such ends as: (a) o e iew he in e en ion ounda ion, indica ing possible di ec ions o ac ion, which Le i on and Hughes (1981) called concep ual use, and C onbach (1982) and Weiss and Bucu alas (1981) enligh enmen ; and (b) o s i up a poli ical deba e abou some social p oblem; o example, he AIDS p e en ion p og ams p omo ed by he local adminis a ions we e no e y e ec i e o his pu pose, bu hey we e good o se ing o a social deba e abou young people’s new ways o expe iencing sexuali y. This has been called a symbolic o poli ical use (Le i on & Hughes, 1981). None o hese no ions was conce ned wi h iden i ying he u iliza ion wi hin a concep ion based on he e y p ocess o e alua ion, which is he cen al axis o he de elopmen al e alua ion p oposed by Pa on (1994). This e alua ion can be conside ed a p ocess o cons an dialogue be ween e alua o s and he pa ies in ol ed, who sha e he esponsibili y o gene a e, ansmi , and use he in o ma ion. E e y hing p oduces lea ning in indi iduals du ing he e alua ion p ocess, ha is, a p ocess o cons an eedback and change, whe e he e alua o wo ks wi h a eam o pe sons who wan o lea n how o imp o e wha hey a e doing and o lea n om hei ac ions. Pa on (1997) was also in e es ed in he in en ional na u e o he knowledge gene a ed by he e alua ion, so his p oposal includes he iden i ica ion and di e en ia ion o he p ima y use s om he audience in gene al, as hese elemen s may p e e di e en usages o he e alua ion. Fo example, a poli ician, as p ima y use , may be in e es ed in he esul s o he e alua ion o jus i y his s ance ega ding some social p oblem, whe eas he ci izens may be in e es ed in he use o he esul s o he e alua ion o imp o e he policy o dis ibu ion o goods. All he abo e indica es he mul idi ec ional na u e o he usage o he esul s o e alua ion, which has led Ki kha (2000) o p opose ha i may be p e e able o alk abou in luence ins ead o use, so as o unde line he unin en ional and nonins umen al e ec s o he e alua ion. Fo his au ho , i is essen ial o de e mine he sou ces o in luence o such elemen s o he e alua ion ha comp ise he ac i e agen s o change and ha may be associa ed bo h wi h he p ocesses and he esul s o he e alua ion, and also he empo al dimension, o momen in which he in luence occu s, because i s e ec s will a y depending on whe he i occu s du ing o a he end o he e alua ion, o a some in e al a e he comple ion o he e alua ion. In sho , i is ob ious ha esea ch o he usage has aken a new di ec ion in he las yea s, going om being a mino issue o becoming a undamen al aspec in esea ch. This is no su p ising i one akes in o accoun ha e alua i e wo k should mee speci ic equi emen s bo h o he poli ical and he scien i ic con ex , equi emen s which a e cu en ly seen as in e wo en, hus sha e ing he a ional ou look o he u iliza ion o he esul s o social expe imen a ion, which had been he p edominan iewpoin since he decade o he six ies (Campbell, 1988). An In eg a i e App oach A e e iewing he componen s o he e alua ion, we p opose an o ganizing guideline in which he a ious ways o pe o ming an e alua ion can be in eg a ed. In ou app oach, he di e en combina ions o hese elemen s a e implici in he goals o he e alua ion, and hese combina ions gene a e di e en ways o add ess he dimensions o e alua ion, as shown in Table 1. Acco ding o his guideline, he e alua o could adop one o se e al o he ollowing combina ions, depending on he goals o he e alua ion: 1. Planning in e en ions. 2. Lea ning and con inuous imp o emen . 3. P og amming policies. 4. T ans o ma ion o socie y. Planning In e en ions E alua ions wi hin he amewo k o his goal a e mean o ob ain in o ma ion abou he esul s achie ed by he de elopmen o a p og am, an ac i i y, e c. An example o his ype o e alua ions is he planned a ia ion e alua ion, in which di e en a ie ies o a p og am a e compa ed o de e mine hei di e en ial e icacy (see, o example, Yeh, 2000). The goals o hese e alua ions de ine a mic o pe spec i e and a linea change s a egy; ha is, he e alua ion ocuses on he e aluand, wi h he aim o ob aining in o ma ion abou he e icacy, e iciency, o e ec i eness on which o base a decision abou u u e planning. This means elimina ing ac ions ha ha e p o en o be ine icien o implemen ing ac ions ha ha e been shown o be e icien . This ype o e alua ions, suppo ed by me hodological igo as a quali y c i e ion, equi es he objec i i y and independence o he e alua o , who is esponsible o he en i e e alua ion p ocess. When accep ing he s anda ds o objec i i y and independence, he in o ma ion p o ided by he e alua ion is expec ed o be ele an o he selec ion and adop ion o he bes solu ions, al hough o he unin en ional in luences may co-exis . In his case, in addi ion o imp o ing he e icacy o a p og am, he e alua ion may be use ul o o he la en pu poses, such as inc easing he isibili y o he p og am in he eyes o he communi y. Lea ning and Con inuous Imp o emen The pu pose o his ype o e alua ions is lea ning and con inuous imp o emen o he in e en ion. These e alua ions ocus on he analysis o he co espondence be ween wha EVALUATION OF SOCIAL INTERVENTION 27 was in ended and wha was ac ually implemen ed so as o ca y ou he necessa y adjus men s o p oduce he desi ed change in he goal o he e alua ion. The e o e, hey adop a mic o pe spec i e and a nonlinea s a egy. An example o his ype o e alua ions is he abo e- men ioned heo y-d i en e alua ion, which consis s o he o mula ion o a heo y ha desc ibes he e aluand’s s uc u al and ope a ional cha ac e is ics in de ail, in o de o p o ide in o ma ion abou i s na u e, unc ioning, and aspec s ha may be subjec o change. Fo a e iew o he mos ecen applica ions o his model, see Donaldson (2003). Con a y o he p e ious pe spec i e, his ype o e alua ion has a o ma i e na u e and he pa icipa ion o hose esponsible o he implemen a ion and ini ia ion o he p og am being e alua ed is essen ial. They should play a ele an ole in he speci ica ion o he e alua ion issues, collec ing in o ma ion, in e p e ing he esul s, e c. This pa icipa ion has ob ious e ec s on he suppo o he p og am and on i s e alua ion, di ec ly a ec ing decision making and p oblem sol ing. The s ance ha he e alua o should adop abou he alues o he g oups in ol ed can a y om dis an o nea . In he i s case, he e alua o jus compiles, desc ibes, and examines hese alues neu ally. In he second case, he e alua o ies o unde s and he meaning o he p og am o he pe sons in ol ed, because his meaning e eals which aspec s o he p og am should be changed o imp o e i s unc ioning. Independen ly o he e alua o ’s s andpoin , he o she wo ks in collabo a ion wi h he g oups in ol ed, mode a ing and cla i ying he discussion, acili a ing decisions on he basis o he da a ob ained. Las ly, om his pe spec i e, he me hodological choices will depend on he na u e o he in e en ion and on he deg ee o speci ica ion and o maliza ion o he in e en ion. Table 1 Guideline o he E alua ion o Social In e en ion P og ams Goal o he E alua ion Dimensions E aluand E alua ion c i e ia Le el o pa icipa ion E alua o ’s s andpoin E alua o ’s esponsibili y E alua o ’s ole Scien i ic assump ions Me hods o conside a ions o he design o he e alua ion In luence o he e alua ion Planning in e en ions (mic o pe spec i e, linea change) P og ams, in e en ions o ac ions pe o med E icacy, e iciency o e ec i eness The audience is in o med o consul ed Independence, objec i i y Maximum Scien i ic S anda ds ha accoun o causali y Expe imen al, quasi-expe imen al, o co ela ional Focused on e alua ion esul s. P omo ed a he end o he e alua ion. Lea ning and con inuous imp o emen (mic o pe spec i e, nonlinea change) De elopmen o he p og am, ac ion, o in e en ion Quali y and implemen a ion Pa icipa ion in pa o o in he en i e p ocess Examines he alues Sha ed wi h he in ol ed g oups Mode a o Emphasis on he speci ica ion o he con ex Eme ging designs, mixed designs, case s udies Focused on e alua ion p ocess. P omo ed om he s a . P og amming policies (mac o pe spec i e, linea change) Se ies o in e en ions wi h common goals Impac The audience is in o med o consul ed Independence, objec i i y Maximum Resolu i e Sea ching o gene alizable conclusions ega ding he in luence o he con ex Me a-analysis. Syn hesis. Good p ac ice Focused on e alua ion esul s. P omo ed a he end o he e alua ion. T ans o ma ion o socie y (mac o pe spec i e, nonlinea change) Ac ions ha lead o disc imina ion and inequali y Social jus ice and inclusion Ac i e pa icipa ion. De ense o alues Minimal P omo e He meneu ical cycle o in e p e a ion Dialogue. C i ical sel - e lec ion Comp ises he inal goal o he e alua ion Thus, o example, when p oblems a e no well de ined, people disag ee abou which aspec s should ha e p io i y, he goals a e di use, o he ac i i ies a e no implemen ed in hei o ali y, hen he e alua o should s a ou wi h na u alis ic assump ions so as o disco e and unde s and he phenomena being s udied in hei con ex . Acco ding o his iewpoin , i is no necessa y o begin wi h a p io design, bu ins ead a design “eme ges” and akes shape as he e alua ion ad ances; quan i a i e measu es can subsequen ly be inco po a ed o inc ease he c edibili y o he indings (see o example Kala a & Illback, 1998). On he con a y, when he p og am is highly speci ic and o malized, one should choose quan i a i e me hods om he s a . P og amming Policies In his g oup a e he e alua ions whose pu pose is o o mula e gene al in e en ion p inciples o help imp o e social policies and compile he bes p ac ices o lessons lea ned. In his case, he e alua ions go beyond he limi s o a conc e e p og am o ac ion, adop ing a mac o pe spec i e aimed a poli ical, social, o ganiza ional, and psychological ac o s ha ha e acili a ed o hinde ed he achie emen o he bes esul s (linea s a egy). The e a e wo op ions he e: he i s consis s o pe o ming a me a-analysis o he indi idual wo ks and in eg a ing he indings ob ained by each o he pa icula e alua ions and e ealing e ec s ha would o he wise no be obse able (see, o example, Lipsey, 1997; Lipsey & Wilson, 1993). The second op ion consis s o compa ing a se ies o in e en ions wi h common aims and cha ac e is ics bu aking place in di e en con ex s. This ype o e alua ion is known as clus e e alua ion (see Sande s, 1997). The pu pose o bo h ypes o s udies is o ob ain gene alizable indings ha lead o be e comp ehension o he in e en ion p og ams by analyzing he di e se ci cums ances and con ex s in which hese in e en ions achie e he bes esul s. F om his iewpoin , he e alua o s do no limi hei ole o ha o me e ad iso s in he decision-making p ocess o de elop new ini ia i es, policies, o implemen a ion s a egies, bu ins ead hey also make sugges ions o sol e he p oblems. T ans o ma ion o Socie y As in he p e ious case, his ype o e alua ions a e wi hin he socio-poli ical con ex , bu con e sely, hey exp ess misgi ings abou he main aspec s o he p og ams in o de o p omo e subs an ial changes in alues and poli ical p io i ies (mac o pe spec i e). Thus, hey gene a e a pe manen deba e abou he p ocesses ha e eal he weaknesses o he cu en social sys em (nonlinea change s a egy) so as o sol e he mos u gen social p oblems: po e y, isola ion, delinquency, e c. Fo example, he main pu pose o he eminis e alua ion is o make explici he elemen s ha explain gende inequali y (Sielbeck-Bowen, B isola a, Seiga , Tischle , & Whi mo e, 2002). Fo his pu pose, he e alua o adop s an ac i is s ance ha de ends he in e es s o he g oups in ol ed, in pa icula o hose who lack powe and igh s, and p omo es he ac i e pa icipa ion o hese g oups o ans o m he exis ing s uc u es. Con e sely, he e alua ion is based on deba e, dialogue, and e lec ion wi hin he he meneu ical cycle o obse a ions and in e p e a ions ha lead o he unde s anding and comp ehension o eali y. Las ly, gi en he c i ical examina ion o he alues and assump ions o he p og ams and he need o make he cu en inequali ies explici , he e alua ion plays an ac i e ole in poli ical ac i i y, because he goal is o he e alua ion o be linked o social ans o ma ion. Summing up, he pu pose o his guideline is o he e alua ion o be comp ehensi e and ealis ic; ha is, o accep ha he e alua ion mee s he demands and needs o a ce ain social con ex and he goals o sol ing social p oblems, using he e alua ion heo y ha is mos app op ia e o one’s pu poses. This pe spec i e, a he han b eaking up he ield o e alua ion, should, o a g ea ex en , con ibu e o he p oduc i e p og ess and plu ali y o in e es s, pe spec i es, and alues o socie y. Re e ences Ál a o, J.L. (1995). Psicología social: pe spec i as eó icas y me odológicas. Mad id: Siglo XXI. Al i a, F. (1983). Pe spec i a cuan i a i a-cuali a i a en la me odología sociológica. Re is a española de in es igaciones sociológicas, 22, 53-75. Bickman, L. (2000). Summing up p og am heo y. New Di ec ions o E alua ion, 87, 103-111. Bi ckmaye , J.D., & Weiss, C.H. (2000). Theo y-based e alua ion in p ac ice. E alua ion e iew, 24, 407-431. Campbell, D.T. (1988). Me hodology and epis emology o social science. Selec ed pape s. Chicago: Uni e si y o Chicago. Chelismky, E., & Shadish, W.R. (Eds.) (1997). E alua ion o he 21s cen u y. A handbook. Thousand Oaks, CA: Sage. Chen, H.T. (1990). Theo y-d i en e alua ions. Newbu y Pa k, CA: Sage. Cook, T.D. (1997). Lessons lea ned in e alua ion o e he pas 25 yea s. In E. Chelimsky & W. Shadish (Eds.), E alua ion o he 21s cen u y: A handbook (pp. 30-52). Thousand Oaks, CA: Sage. Cook, T.D. (2000). The alse choice be ween heo y-based e alua ion and expe imen a ion. New Di ec ions o E alua ion, 87, 27-34. C onbach, L.J. (1982). Designing e alua ions o educa ional and social p og ams. San F ancisco: Jossey-Bass. Donaldson, S.I. (2003). Theo y-d i en p og am e alua ion in he new millennium. In S.I. Donaldson & E.M. Sc i en (Eds.), HERRERA, LEÓN, AND MEDINA 28 EVALUATION OF SOCIAL INTERVENTION 29 E alua ing social p og ams and p oblems: Visions o he new millennium (pp. 109-141). Mahwah, NJ: E lbaum. Fe e man, D.M. (2001). Founda ions o empowe men e alua ion. Thousand Oaks, CA: Sage. Guba, E.G., & Lincoln, Y.S. (1989). The ou h-gene a ion e alua ion. Newbu y Pa k, CA: Sage. House, E.R. (1980). E alua ing wi h alidi y. Be e ly Hills, CA: Sage. House, E.R. (2001). Responsi e e alua ions (and i s in luence on delibe a i e democ a ic e alua ion). New Di ec ions o E alua ion, 92, 23-30. House, E.R., & Howe, K.R. (2000). Delibe a i e democ a ic e alua ion. New Di ec ions o E alua ion, 85, 3-12. Julnes, G., & Ma k, M.M. (1998). E alua ion as sensemaking: Knowledge cons uc ion in a ealis wo ld. New Di ec ions o E alua ion, 78, 33-52. Kala a , J., & Illback, R. (1998). A quali a i e e alua ion o school- based amily esou ce and you h se ice cen e s. Ame ican Jou nal o Communi y Psychology, 26, 573-604. Ki kha , K.E. (2000). Reconcep ualizing e alua ion use: An in eg a ed heo y o in luence. New Di ec ions o E alua ion, 88, 5-23. Le i on, L.C., & Hughes, E.F. (1981). Resea ch on he u iliza ion o e alua ions: A e iew and syn hesis. E alua ion Re iew, 5, 525-548. Lipsey, M.W. (1997). Wha can you build wi h housands o b icks? Musings on he accumula ion o knowledge in p og am e alua ion. New Di ec ions o E alua ion, 76, 7-23. Lipsey, M.W., & Co d ay. D.S. (2000). E alua ion me hods o social in e en ion. Annual e iew o Psychology, 51, 345-375. Lipsey, M.W., & Wilson, D.B. (1993). The e icacy o psychological, educa ional and beha io al ea men : Con i ma ion om me a-analysis. Ame ican Psychologis , 48, 1181-1209. Ma k, M.M. (2001). E alua ion’s u u e: Fu o , u ile o e ile? Ame ican Jou nal o E alua ion, 22, 457-479. Ma k, M.M., Felle , I., & Bu on, S. (1997). In eg a ing quali a i e me hods in a p edominan ly quan i a i e e alua ion. A case s udy and some e lec ions. New Di ec ions o E alua ion, 74, 47-59. Me ens, D.M. (2003). The inclusi e iew o e alua ion: Visions o he new millennium. In S. I. Donaldson & E. M. Sc i en (Eds.), E alua ing social p og ams and p oblems: Visions o he new millennium (pp. 91-107). Mahwah, NJ: E lbaum. Pa on, M. Q. (1994). De elopmen al e alua ion. E alua ion P ac ice, 15, 311-319. Pa on, M.Q. (1997). U iliza ion- ocused e alua ion: The new cen u y ex (3 d Ed.). Thousand Oaks, CA: Sage. Pa on, M.Q. (2001). E alua ion, knowledge managemen , bes p ac ices, and high quali y lessons lea ned. Ame ican Jou nal o E alua ion, 22, 329- 336. Rebolloso, E., & Rebolloso, J.R. (1998). Signi icado y desa ollo ac ual de la e aluación de p og amas. In E. Rebolloso (Ed.), E aluación de p og amas: ámbi os de aplicación (pp. 9-29). Ba celona: Tex os Uni e si a ios “Sain Jo di.” Reicha d , C.S., & Cook, T.D. (1979). Beyond quali a i e e sus quan i a i e me hods. In T.D. Cook & C.S. Reicha d (Eds.), Quali a i e and quan i a i e me hods in e alua ion esea ch (pp. 7-32). Be e ly Hills, CA: Sage. Rossi, P.H., F eeman, H.E., & Lipsey, M.W. (1999). E alua ion. A sys ema ic app oach (6 h ed.). Thousand Oaks, CA: Sage. Sande s, J.R. (1997). Clus e E alua ion. In E. Chelimsky & W. Shadish (Eds.), E alua ion o he 21s cen u y: A handbook (pp. 396-404). Thousand Oaks, CA: Sage. Schwand , T.A. (1997). The landscape o alues in e alua ion: Cha ed e ain and unexplo ed e i o y. New Di ec ions o E alua ion, 76, 25-37. Sc i en, M. (1980). The logic o e alua ion. In e ness, CA: Edgep ess. Sc i en, M. (1994). The inal syn hesis. E alua ion P ac ice, 15, 367-382. Shadish, W.R. (1987). P og am mic o- and mac o heo ies: A guide o social change. New Di ec ions o P og am E alua ion, 33, 93-109. Shadish, W.R., Cook, T.D., & Le i on, L.C. (1991). Founda ions o p og am e alua ion: Theo ies o p ac ice. Newbu y Pa k, CA: Sage. Sielbeck-Bowen, K.A., B isola a, S., Seiga , D., Tischle , C., & Whi mo e, E. (2002). Explo ing eminis e alua ion: The g ound om which we ise. New Di ec ions o E alua ion, 96, 3-8. S ake, R.E. (1983). P og am e alua ion, pa icula ly esponsi e e alua ion. In G.F. Madaus, M.S. Sc i en, & D.L. S u lebeam (Eds.), E alua ion models: Viewpoin s on educa ional and human se ices e alua ion (pp. 287-310). Bos on: Kluwe - Nijho . S ake, R.E., & Migo sky, C. (1997). The e ol ing syn heses o p og am alue. E alua ion P ac ice, 18, 89-103. S u lebeam, D.L. (2001). E alua ion models. New Di ec ions o E alua ion, 89, 7-98. Vedung, E. (1993). Modelos de e aluación. Re is a de se icios sociales y polí ica social, 30, 39-68. Veney, J.E., & Kaluzny, A.D. (1991). E alua ion and decision making o heal h se ices p og am (2nd Ed.). Ann A bo , MI: Heal h Adminis a ion P ess. Weiss, C.H. (1997). How can heo y-based e alua ion make a g ea e headway? E alua ion Re iew, 21, 501-524. Weiss, C.H. (1998). E alua ion (2nd Ed.). Englewood Cli s, NJ: P en ice Hall. Weiss, C.H., & Bucu alas, M.J. (1981). T u h es and u ili y es s: Decision-make s ame o e e ence o social science esea ch. E alua ion S udies Re iew Annual, 6, 695-706. Wholey, J.S. (1983). E alua ion and e ec i e public managemen . Bos on: Li le B own. Yeh, S.S. (2000). Building he knowledge base o imp o ing educa ional and social p og ams h ough planned a ia ion e alua ions. Ame ican Jou nal o E alua ion, 21, 27-40. Recei ed: May 26, 2004 Re ision ecei ed: No embe 11, 2004 Accep ed: No embe 30, 2004