Evaluation of intervention programs based on physical activity and sport for drug use prevention: a multi-criteria perspective

Table 1. Objectives associated with dimension Facilities Table 2. Attributes associated with dimension Facilities Table 3. Objectives associated with dimension Experience of the applicant organization Table 4. Attributes associated with dimension Experience of the applicant organization Table 5. Objectives associated with dimension Technical proposal: Personnel Table 6. Attributes associated with dimension Technical proposal: Personnel Table 7. Objectives associated with dimension Technical proposal: Program methodology Table 8. Attributes associated with dimension Technical proposal: Program methodology Table 9. Objectives associated with dimension Technical proposal: Methodology of the intervention sessions Table 10. Attributes associated with dimension Technical proposal: Methodology of the intervention sessions Table 11. Objectives associated with dimension Technical proposal: Evaluation methodology Table 12. Attributes associated with dimension Technical proposal: Evaluation methodology Table 13. Objectives associated with dimension Technical proposal: Dissemination Table 14. Attributes associated with dimension Technical proposal: Dissemination Table 15. Objectives associated with dimension Economic proposal Table 16. Attributes associated with dimension Economic proposal Table 1. Objectives associated with dimension Facilities 1. Number of spaces and multifunctionality This objective refers to the quantity and versatility of the spaces used during project implementation. Having a variety of material resources and spaces can boost the potential of a competent and experienced work team, giving them opportunities to design creative and innovative activities, and achieving greater enjoyment for participants (Becoña, 2023). A sports facility may contain more than one space, understood as the spatial delimitation in a facility, where the physical activity can be developed or the sport played. Due to the growth of amateur and recreational sport, spaces are necessarily multifunctional nowadays (Westerbeek et al., 2006). Multi-purpose spaces or rooms are designed and equipped for the practice of more than one sport or physical activity. 2. Type of management The type of facility management (public and private) will condition the organization’s purposes, its values and its activities (de Andrés, 1997; Westerbeek et al., 2006). The values of public management are based on ensuring that the population has basic access to sport, making services available and accessible to everyone and thus improving the social structure. The values of private management are linked to economic performance as a result of risk-taking in a competitive environment, giving rise to an offer of original and innovative activities for a specific public, with admission requirements (quotas). 3. Administrative formula for space management The administrative management formula refers to the procedures or operating systems typified by current legislation adopted for management purposes (de Andrés, 1997). There are two sports facilities management models or formulas: direct or indirect (Ammon et al., 2004). In direct management, the service is provided by the local authority, public organization or corporation itself. On the other hand, when the organization that manages and controls service provision is not a public authority, the management system is referred to as indirect. In turn, indirect management implies intermediate formulas, such as licenses. It is common for a combination of different management formulas to be used for sports services. However, this decision must be carefully thought out since the management formula will influence now much freedom and decision-making power the organization has with respect to space management. 4. Use of natural spaces The literature on the relationship between the contact with nature and general health suggests that there is a socio-behavioral pathway whereby nature has a restorative effect (Hartig et al., 2014). Being in contact with or living near natural environments or spaces favors the practice of PAS and the development of healthy attitudes, leading to the improvement of indicators related to physical health (Calogiuri & Chroni, 2014). Besides, this has also been proven to have positive psychological effects on different dimensions of mental health (cognitive functioning, emotional wellbeing, stress levels) (Bratman et al., 2019), leading to a lower prevalence of mental health problems (van den Berg et al., 2010). From a pedagogical perspective, natural environments offer an optimal context for educational development at a personal and social level in children and adolescents (Becker et al., 2017; Mann et al., 2022) and can be used as a context for intervention in drug use prevention. Table 2. Attributes associated with dimension Facilities 1. Number of spaces and multifunctionality Type of attribute Nominal discrete IR No EA No Values a) 1 multipurpose space, b) 1 non-multipurpose space, c) more than 1 non-multipurpose space, d) more than 1 multipurpose space, e) 1 multipurpose space and 1 nonmultipurpose space, f) at least 1 multipurpose space and more than 1 non-multipurpose space. 2. Type of management Type of attribute Nominal discrete IR No EA No Values a) publicly owned spaces, b) privately owned spaces, c) mixed (a combination of publicly and privately owned spaces). 3. Administrative formula for space management Type of attribute Nominal discrete IR No EA No a) direct management, b) licenses, c) indirect management, d) direct management and licenses (a+b), e) direct management and indirect management (a+c), f) licenses and indirect management (b+c). 4. Use of natural spaces Type of attribute Binary IR No EA No Values a) Yes, b) Not. Information will be requested (IR); Expert assessed (EA) Table 3. Objectives associated with dimension Experience of the applicant organization 1. Proximity of the organization to the program's target population It may be advantageous for an organization to be based in and be part of the social network of the neighborhood if it is to work on prevention objectives. It will be more connected to the target population and will be acquainted with the needs and problems of the neighborhood in relation to drug use (Brotherhood & Sumnall et al., 2019), possibly being part of a team or community coalition fighting drug use (EMCDDA, 2019, 2022b), or being coordinated with other programs or efforts that are trying to prevent drug use (CCSAb, 2010). This, of course, will be rated positively. 1.1. Community presence Regarding geographical scope, the organization that is responsible for designing the intervention may not be the same as the organization that is in charge of implementing the program. 1.2. Collaborations with the neighborhood Collaborations with the neighborhood refer to previous collaborations that the applicant organization has had with other local organizations in the neighborhood: local businesses, sports centers, social services, neighborhood or festival associations, neighborhood police, etc. Note that these collaborations only consider organizations that do not deal with drug use (prevention, treatment, rehabilitation) or are not involved in social intervention generally (organizations dealing with drug use or involved in social intervention will be considered afterwards). Note also that collaboration means carrying out activities involving both organizations and whose recipients are the program participants, excluding relationships based on the exchange, donation or transfer of material resources, spaces or economic resources. 2. Specialization in statutes The purpose for which the organization was established stands out above the other aspects included in its statutes. It will be positively evaluated if the organization was set up for the specific purpose of preventing drug use among adolescents. 3. Previous experience of the applicant organization An organization is assumed to be experienced in the issue when it has already implemented drug use prevention projects and, consequently, has managed to establish collaborations with other stakeholders to achieve prevention (EMCDDA, 2019). Therefore, previous experience will make the organization better qualified to implement the intervention program and the decisions that need to be made during its development. 3.1. Previous addiction prevention projects Regarding previous addiction prevention projects/programs, only projects targeting addiction prevention structured to implement at least two sessions per week for three months will be taken into account (projects or interventions based on informative sessions, like conferences, will be excluded). To accurately evaluate the organization’s experience in the implementation of projects/programs, the organization will be asked to provide information on the number of projects, the type of call for proposals through which they were funded, their duration and the funding received. 3.2. External collaborations Regarding external collaborations, the applicant organization has been able to establish links with different public or private organizations (universities, NGOs, social services, private companies) to address the problem. As for the previous criteria, collaboration is understood as the performance of activities in which both parties are involved targeting the project participants, excluding relationships based on the exchange, donation or transfer of material resources, spaces, or economic resources. Additionally, only collaborations with organizations working on the problem of drug use and addictions or in social intervention generally will be considered. Information will be requested on the number of organizations with which the applicant organization has collaborated (over the last 10 years), the ownership of these organizations and the duration of the collaboration. Table 4. Attributes associated with dimension Experience of the applicant organization 1. Proximity of the organization to the program's target population 1.1. Community presence Type of attribute Nominal discrete IR No EA No Values a) the applicant organization and the organization that is to implement the program belong to the geographical scope of the district or neighborhood where the program is to be implemented / the organization that is to design and implement the program belongs to the geographical scope of the district or neighborhood where the program is to be implemented, b) the applicant organization belongs to the geographical scope of the district or neighborhood, whereas an external organization is to develop the program, c) the applicant organization is external to the district or neighborhood, whereas the organization that is to develop the program falls within its geographical scope, d) the applicant organization and the organization that is to develop the program are external to the geographical scope of the district or neighborhood / the organization that is to design and implement the program does fall within the geographical scope in which the program is to be implemented. 1.2. Collaborations with the neighborhood Type of attribute Nominal discrete IR No EA No Values a) 1 organization, b) 2 organizations from the same area, c) 2 organizations from different areas, d) 3 organizations from the same area, e) 3 organizations from different areas, f) 4 organizations from the same area, g) 4 organizations from different areas, h), 5 organizations from the same area, and i) 5 organizations from different areas. * The applicant will be asked to provide the name of each organization by way of information. This will not be used for rating or scoring purposes. 2. Specialization in statutes Type of attribute Binary IR No EA No Values a) Yes, b) Not. 3. Previous experience of the applicant organization 3.1. Previous addiction prevention projects § Number of projects Type of attribute Ordinal discrete IR No EA No Values No projects, 1 project, 2 projects, 3 projects and 4 or more projects. § Type of funding Type of attribute Binary IR Yes EA Yes Values a) Yes, b) Not. § Duration Type of attribute Continuous IR No EA No Values A values ranging from 48 and 200 sessions will be used, i.e., a minimum of 48 sessions is required, whereas 200 sessions are enough to achieve the maximum score for this criterion. § Funding received Type of attribute Continuous IR No EA No Values A values ranging from €5,000 to €50,000, i.e., a minimum of €5,000 is required and €50,000 is enough to achieve the maximum score for this criterion. 3.2. External collaborations § Number of organizations Type of attribute Discrete IR No EA No Values 0, 1, 2, 3, 3, 4 or more. § Type of organizations Type of attribute Nominal IR No EA No Values Public, private or mixed (a combination of the above). § Duration Type of attribute Continuous IR No EA No Values A values ranging from 0 to 60 months will be used, where 60 months are enough to achieve the maximum score for this criterion. Information will be requested (IR); Expert assessed (EA) Table 5. Objectives associated with dimension Technical proposal: Personnel 1. Employees involved in program implementation The program size will determine the number of tasks and profiles needed for its implementation. Whenever possible, staff members should be organized in work teams to cover all project needs (Brotherhood & Sumnall et al., 2011). To evaluate this criterion, professional profiles were classified into three types: A) Essential profiles. Profiles in charge of the development of the sessions and the intervention with the participants, without which the program could not fulfill its main function. The qualifications for the essential profiles include a bachelor’s degree in physical activity and sport sciences (PAS) or a bachelor of education in physical education, a master’s degree or PhD in PAS, an advanced diploma in teaching and socio-sports animation, an advanced diploma in physical conditioning, an advanced diploma in physical and sports activities animation and a technical officials certificate awarded by the respective federation; B) Desirable profiles. Profiles that collaborate with the essential profiles, complementing their work or taking care of functions other than the implementation of the sessions. While they are not essential, their participation adds significant value to the professional team, contributing to the effective development of the program. The following profiles are considered: a bachelor’s degree in psychology or social education, a master’s degree in psychology or social education, a bachelor’s degree in medicine, nursing, marketing, advertising, business administration and management and law; C) Complementary profiles. Social organization or project profiles that are less relevant than the above and whose absence would not affect the development or outcome of the program. The profiles considered are leisure and free time monitor, nursing assistant, others. Regardless of their profile, the members of the work teams must have an adequate set of competencies (training, knowledge and skills) and previous experience (similar projects) (Brotherhood and Sumnall et al., 2011). To evaluate the members of the work teams for each of the profiles, the following will be assessed: the number of workers involved, the quality of their training, and the quality of their experience (participation in drug use prevention projects). 2. Specific program training Table 9. Objectives associated with dimension Technical proposal: Methodology of the intervention sessions 1. Intervention design Session design An intervention composed of a specific number of sessions will be designed to meet the prevention objectives. The sessions should include the specific objectives set out in the program, as well as the resulting contents. A series of activities based on scientific evidence regarding prevention or behavior change that are potentially attractive, interesting and meaningful to participants will be used to work on the objectives and contents of each session (Schinke et al., 2002). For this purpose, different behavior change techniques will be used (Abraham & Michie, 2008; Griffin & Botvin, 2010; Klamert et al., 2023; Michie et al., 2011). Although these techniques are applied in a classroom setting, they can also be applied in the sport context by adapting the sport practice and/or using the sport elements (Collingwood, 2000; Elliot et al., 2004; Goldberg et al., 1996). Another noteworthy aspect is the intervention session structure, which may include dynamics to raise the interest and motivate participants, encourage participation, a positive group climate and group cohesion, or session closure dynamics (CCSA, 2011; EMCDDA, 2019). • Age-sex/gender-addictive substance-ethnicity adjustment Drug use is a potentially very wide-ranging phenomenon depending on both the type of drug consumed and the characteristics of the consumer (age, sex, gender, ethnicity or risk group). The interaction between these factors can influence: the age of onset of consumption, the type of drug consumed, the ritual of consumption, the form of acquisition, beliefs and perception of risk, the effects, the major risks and protective factors or willingness to ask for help (EMCDAA, 2022; SAMHSA, 2023; UNODC, 2023). Ethnicity, cultural influences or sexual orientation are other characteristics of the target population that will help to better define the consumption phenomenon (McCabe et al., 2009). These factors and how they interact with each other should be identified to design successful prevention programs and avoid participant dropout, as well as to tailor the intervention to the target population insofar as possible (Brounstein et al., 2001; EMCDDA, 2019; UNDOC & WHO, 2018). • Variety of physical activity and sports (PAS) performed A diversified, specialized, and novel offer of activities will be more attractive to participants, increasing motivation and interest in the program. From the sports point of view, variety can be understood as the practice of different sports, avoiding monotony and covering the preferences of all participants. Non-traditional or alternative sports are new sports that stand out as being inclusive and motivating (Hall et al., 2016), encouraging the participation of as many people as possible regardless of their ability, and as being contexts that favor the development of mental health and the integral formation of the individual (Caldarella et al., 2024). They contrast with traditional sports, which are oriented towards performance, individualism and competition. Some examples are Kin-Ball or pickleball. • Activities in natural environments A previous criterion evaluated the use of natural areas, this criterion will evaluate the performance of activities/sessions in natural environments. 2. Complementary activities • Special activities Special activities should provide a break from the routine, having a novelty value that motivates participants to continue with the program. They can be purely recreational or include educational content and are usually carried out outside the spaces commonly used to implement the program. • Activities with neighborhood organizations The program can include activities with neighborhood organizations that connect the target population with neighborhood or community organizations. Elements such as rebelliousness, negative relationships with adults, rejection employment opportunities or detachment from the community increase the likelihood of drug use. On the contrary, the generation of opportunities for prosocial and community engagement, together with community recognition of such involvement will promote a feeling of social integration associated with a decrease in consumption among participants (Brounstein et al., 2001; Hawkins et al., 1992). Activities focusing on these protective factors respond to the participants' longing to contribute to their neighborhoods and be part of a community of people with similar values (Ferris et al., 2021). On the other hand, the mobilization of neighborhood organizations connected to the program will help build community coalitions and facilitate the creation of prevention strategies (EMCDDA, 2022b). Creating strong ties between the target population and the community will help to generate a healthy relationship between the two, fostering integration, and avoiding exclusion and stigma (Robertson et al., 2003). Some examples would be to draw attention to local businesses in the neighborhood and participate in some of their activities, offer scholarships to join neighborhood sports teams or involve the target population in the organization of cultural or festive events in the neighborhood. Table 10. Attributes associated with dimension Technical proposal: Methodology of the intervention sessions 1. Intervention design § Session design Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. § Age-sex/gender-addictive substance-ethnicity adjustment Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. § Variety of physical activity and sports performed Type of attribute Binary IR No EA No Values a) Yes, b) Not. * An attribute establishing whether or not at least 20% of the sessions include alternative sports will be used to evaluate this criterion. § Activities in natural environments Type of attribute Binary IR No EA No Values a) Yes, b) Not. * An attribute establishing whether or not at least 20% of the sessions take place in natural environments, leading to the corresponding binary attribute. 2. Complementary activities § Special activities Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. § Activities with neighborhood organizations Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. Information will be requested (IR); Expert assessed (EA). Table 11. Objectives associated with dimension Technical proposal: Evaluation methodology 3. Types of assessment and evaluation techniques • Baseline or needs assessment This assessment provides information on the drug use-related needs of the target population and the factors that contribute to these needs. With the assessment, unmet needs will be detected, thus justifying the intervention. The European standards highlight the needs defined by Brotherhood & Sumnall (2011): 1) policies and legislation on drug dependence, 2) the community where the project is to be implemented, and 3) the target population that is to participate in the project. At the policy and legislation level, the project can be aligned with national drug prevention strategies. At the community level, statistical databases on drug use and other cross-cutting issues (inequality, socioeconomic and socio-educational level) can be consulted. At the target population level, the focus will be on risk and protective factors (occurrence, intensity, grouping by similarity, addressability), target population risk level (universal, selective, indicated), culture and daily life in relation to drug use, etc. • Outcome appraisal When an outcome appraisal is conducted, it should cover all outcomes as defined in the planning phase (i.e., according to previously defined evaluation indicators) (Kröger et al., 2012). Depending on the scope of the program and the chosen research design, statistical analyses should be conducted to determine the effectiveness of the intervention in achieving the defined objectives. Whenever possible, an informal statement on the effectiveness of the intervention, summarizing the findings of the outcome appraisal, is desirable. • Satisfaction evaluation Besides, the evaluation of program participant, staff member and collaborating organization satisfaction can help to identify strengths and weaknesses and establish future improvements at different levels. Note, however, that the satisfaction evaluation is not an acceptable indicator of the effectiveness of the intervention, nor does it provide information on the effects of the intervention on the participants (Brotherhood & Sumnall, 2011). • Evaluation techniques The instruments must be appropriate for measuring the expected indicators (EMCDDA, 2010). Some commonly used instruments are questionnaires, interviews, physiological tests, and observation instruments. Other aspects to consider when selecting assessment instruments are objectivity, reliability and validity (Kröger et al., 2012). 4. Monitoring and control plan Evaluations can be conducted throughout program implementation (EMCDDA, 2019). Monitoring implementation through regular progress reviews helps identify adherence to the original plan and the need for modifications (Brotherhood & Sumnall, 2011). This evaluation answers the question: is the program performing as planned? Monitoring can check that the sessions are being developed according to the planned schedule, there is coordination between agents, and the adaptation of the intervention to the target population is correct, etc. Table 12. Attributes associated with dimension Technical proposal: Evaluation methodology 1. Types of assessment and evaluation techniques § Baseline or needs assessment Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. § Outcome appraisal Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. § Satisfaction evaluation (applied to participants, collaborating organizations and staff members) Type of attribute Binary IR No EA No Values a) Yes, b) Not. § Evaluation techniques Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. 2. Monitoring and control plan Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. Information will be requested (IR); Expert assessed (EA). Table 13. Objectives associated with dimension Technical proposal: Dissemination 1. Participant recruitment Recruitment refers to the process of selecting eligible individuals from the target population, informing potential participants about the program, inviting them to participate, enrolling them and assuring that they start the intervention. Different strategies can be developed to improve recruitment (Powell et al., 202): a) detect and remove any barriers to participation, b) promote participation and increase participant engagement by designing challenging and entertaining activities, c) use social networks to connect with the target population, d) select participants in a methodologically correct and ethical manner. 2. Advertising on ICTs Information and communication technologies (ICT) include both social networks and media specialized in a specific area or specific to a place (locality, neighborhood, province): television channels, radio programs or digital press. The social organization can generate trust in the community by creating added value through social networks (Ferris et al., 2021). Posting content related to the social program on social networks can make participants feel part of a community with shared values and generate an interest in contributing to their respective neighborhoods. This criterion refers to the publication of all content related to the program except for the results, which will be discussed under another criterion. 3. Social, professional and scientific dissemination of results to society environment Once the intervention program has been completed, the results and conclusions of program performance have to be communicated. To do this, it is important to take into account which people are potentially interested in the results, what information is most relevant and what forms of communication will be used (Brotherhood & Sumnall, 2011; Kröger et al., 2012). Under this criterion, a distinction has been made between dissemination to either the general public or to the professional and scientific community. Dissemination to the general public can benefit the program in many ways, for example, by garnering the support of key stakeholders to assure program continuity or improving the program through feedback; dissemination to specialized fields contributes to improving the design of future prevention practices and research or facilitating the replication of the intervention by others. Table 14. Attributes associated with dimension Technical proposal: Dissemination 1. Participant recruitment Type of attribute Ordinal discrete IR Yes EA Yes Values None, low, moderate and high. 2. Advertising on ICTs Type of attribute Nominal IR No EA No Values a) no publicity plan, b) publicity plan involving one or two ICTs (not carried out by a community manager), c) publicity plan for three or more ICTs (not carried out by a community manager), d) publicity plan carried out by a community manager. 3. Social, professional and scientific dissemination of results to society environment § Dissemination to the general public Type of attribute Nominal IR No EA No Values a) no publicity plan, b) publicity plan involving one or two ICTs (not carried out by a community manager), c) publicity plan for three or more ICTs (not carried out by a community manager), d) publicity plan carried out by a community manager. § Dissemination to specialized fields (applied to: a) dissemination media in specific professional fields, b) communication at conferences, congresses, symposia, etc., c) specialized journals without impact, d) specialized journals with SJR or JCR relative impact index) Type of attribute Binary IR No EA No Values a) Yes, b) Not. Information will be requested (IR); Expert assessed (EA). Table 15. Objectives associated with dimension Economic proposal 1. Applicant organization solvency Projects should be designed to ensure their sustainability over time (Brotherhood & Sumnall, 2011), that is, the project should not require further funding to continue in the future. Some indicators of access to funding possibly demonstrating the long-term solvency of an organization are: 1) a manager specialized in the task, 2) strategies to raise additional funding, 3) funding derived from a wide range of sources, and 4) the organization holds quality accreditations and certificates. 2. Budget consistency The financial planning needs (costs) and capacities (budget) play a key role in the economic proposal (Brotherhood & Sumnall, 2011). It must be possible to achieve the set objectives, without going over budget (Becoña, 2023). Likewise, the budget and expenses must be consistent with the program design: proposed activities, personnel, spaces, etc.