Patrimonializarte: A heritage education program based on new technologies and local heritage

education sciences Article Patrimonializarte: A Heritage Education Program Based on New Technologies and Local Heritage Leticia Castro-Calviño 1,* , Jairo Rodríguez-Medina 2, Cosme J. Gómez-Carrasco 3and Ramón López-Facal 1 1Department of Applied Didactics, Faculty of Education Sciences, University of Santiago de Compostela, 15782 Santiago de Compostela, Spain; [email protected] 2Department of Pedagogy, Faculty of Education, University of Valladolid Campus Miguel Delibes, Paseo de Belén, 1, 47011 Valladolid, Spain; jairo.r[email protected] 3Department of Mathematics and Social Sciences Teaching, Faculty of Education, University of Murcia, 30100 Murcia, Spain; [email protected] *Correspondence: leticiacastr[email protected] Received: 26 May 2020; Accepted: 28 June 2020; Published: 1 July 2020   Abstract: This paper presents the design of the Patrimonializarte program, which has been implemented in six classes and three different educational levels in two schools in Galicia (Spain). It involves working with elements of heritage in proximity to the schools’ pupils by employing Information and Communication Technology (ICT) tools. The objectives are (a) to find out if the design is coherent and relevant according to expert judgement and (b) to discover whether the activities related with new technologies are effective according to the evaluation of those involved in the program. This is evaluation research employing mixed methods in which a collective study of cases is carried out. The main results show (1) the evaluation of the design and the tools of the program achieve an optimal degree of agreement according to the panel of experts as far as the variables measured are concerned, and (2) the schoolchildren and teaching staffprovide a positive evaluation of the use of ICT tools for working with heritage. E-learning and m-learning make it possible to motivate pupils in the learning and teaching process. Working with ICT tools acquires importance with regard to the possibilities they offer to disseminate the heritage. An integral evaluation of programs is considered relevant in order to understand their multi-dimensionality. Keywords: technological resources; e-learning; heritage teaching; program evaluation; evaluation methods; design evaluation 1. Introduction This paper originates from a prior exploratory study, which identified the needs of infant, primary, and secondary teachers in relation to working with heritage in the classroom [ 1 ]. The results obtained reflected the interest of teachers in innovative activities including the use of ICT tools, such as the creation of videos and the gathering of information regarding different heritage elements in order to put it at the disposal of the general public. The teachers identified the need to have at their disposal education programs that help them to work with heritage, as they recognise that their training was deficient in this area. With the aim of responding to these needs, the heritage education program Patrimonializarte was designed and adapted to the context and the circumstances in which it was to be carried out. The teachers stated that their pupils did not recognise their local heritage and that they demonstrated a lack of motivation in working on it. The program focuses on working with the heritage in the pupils’ local area and employs innovative methodologies, including information and communications Educ. Sci. 2020,10, 176; doi:10.3390/educsci10070176 www.mdpi.com/journal/education Educ. Sci. 2020,10, 176 2 of 19 technology. The aim is to enable pupils to identify their heritage, to value it, and to contribute towards its dissemination and preservation. In this way, an impact on the community can be achieved, which helps to create individual and shared identities, thereby contributing towards the safeguarding of the local heritage. 1.1. The Evaluation of Heritage Education Programs Heritage education has acquired increasing importance in the education system in Spain. Two of the institutional points of reference in heritage education in Spain are the Plan Nacional de Educaci ó n y Patrimonio (National Plan for Education and Heritage) (PNEyP), focused on research, innovation, and training in heritage, managed by the Instituto del Patrimonio Cultural de España (Cultural Heritage Institute of Spain) and dependent on the Spanish Ministry of Education in collaboration with the regional governments, and the Observatorio de Educaci ó n Patrimonial en España (OEPE) (Observatory of Heritage Education in Spain), directed by Olaia Fontal (University of Valladolid). The OEPE manages a database which inventories, analyses, and evaluates heritage education programs [ 2 , 3 ]. Since 2010, it has included more than 1200 references. However, only 1.9% of these programs envisage some kind of evaluation tool [ 3 ]. This is a significant limitation or deficiency, given that evaluation should be an essential part of any education program. The OEPE has carried out analyses and categorisations of these heritage education programs [ 4 – 7 ] and has, recently, created a tool for the evaluation of the quality of heritage education programs; the Q-Edutage scale based on evaluation by standards [ 8 ]. However, the evaluation of heritage education continues to be present in a limited number of education programs [ 9 ]. Thus, there is a need to broaden research in this field in order to be able to establish guidelines and models that contribute towards an improvement in the quality and effectiveness of these programs. 1.2. An Evaluation of New Technologies in Heritage Education The acquisition of digital competence is one of the capacities that are established, and commonly accepted, in education around the world. To a greater or lesser degree, most schools include work with ICT in their classrooms [10]. Pupils can go from being mere consumers of information via the Internet to also being producers of content and active subjects in our information society. There are many software programs such as Eduloc, Map My Tracks, and Google Earth, among others, which make it possible to georeference and geolocate contents and points of interest in order to share information with the rest of the digital community. The contents generated can be viewed on mobile devices with free applications. These resources facilitate the introduction of mobile technology into learning environments, which is known as mobile learning [ 11 ]. One advantage of this type of software is that students can establish any element as a point of interest. Not only can material or monumental elements be referenced, but other immaterial elements can also be included, which can prove extremely useful in promoting the conservation and dissemination of intangible heritage. There is already a broad literature on the use of ICT (e-learning and m-learning) in heritage education. The majority is in agreement regarding the great potential of these tools for working with heritage [ 12 , 13 ]. There are also many references to program evaluation in heritage education with a technological basis, both in the field of education and in museums [ 14 – 17 ]. However, program evaluation in heritage education is a field that has not yet been developed as it should [ 18 ]. There is a lack of integral evaluations considering program evaluation from the beginning (analysis of needs, design, initial evaluation), during its implementation (formative evaluation), and at its end (summative evaluation and meta-evaluation). Therefore, the development of evaluation models including these aspects is considered a necessity. However, what type of model would be appropriate for heritage education evaluation programs? Educ. Sci. 2020,10, 176 3 of 19 1.3. An Approach to Evaluation Research in Education There is a certain degree of consensus on the establishment of different stages in education research and, therefore, in program evaluation. At present, program evaluation has moved away from the phase prior to tests and measurement, to which Lukas and Santiago [ 19 ] make reference, and from the stage of tests or effectiveness, which took place at the beginning of the 20th century, which Guba and Lincoln call The First Generation, or the measurement generation [20]. Beginning in 1942, what is considered to be the first systematic and rigorous study of program evaluation was carried out. Ralph Tyler studied 30 schools over a period of eight years (the Eight-Year Study), during which he carried out a program evaluation in order to analyse the effectiveness of the curricula. Tyler proposed a model of educational evaluation that emphasised the achievement of objectives, the organisation of the curriculum based on these objectives and providing value judgements on the information gathered, thereby moving away from measurement [21]. During the 1960s, the learners were the subject of evaluation. This became known as the age of description, what Guba and Lincoln call The Second Generation [ 20 ]. At the end of the 1960s, there was a stage known as Accountability when the evaluation was oriented towards the individual. A period of theoretical production began regarding the multiple dimensions of the evaluation process, which would come to be known as evaluation research [ 22 ]. Two papers by Cronbach and Scriven changed the direction of this field and focused attention on evaluation oriented towards decision-making [ 23 , 24 ]. In accordance with this, in the 1970s, there was a proliferation of evaluation models. There was a move to what Guba and Lincoln call The Third Generation: Judgment [ 20 ], including models such as the decision-oriented model: Context, Input, Process and Product (CIPP) by Stufflebeam et al. [ 25 ] and Alkin’s Center for the Study of Evaluation (CSE) [26]. From 1973 onwards, evaluation became consolidated as a specific field of research (the stage of professionalisation). Alternative models began to appear: Scriven’s Goal Free Model [ 27 ]; the responsive evaluation of Stake [ 28 ]; MacDonald’s democratic evaluation [ 29 ]; Parlett and Hamilton’s model of illuminative evaluation [ 30 ]; and Eisner’s evaluation as artistic criticism [ 31 ]. Journals specialising in evaluation began to be published and specific subjects on evaluation began to be taught in the great universities of the United States. Professional associations were established, such as the Joint Committee on Standards for Educational Evaluation (1975), which publishes the guidelines for the evaluation of programs, projects and education materials [32]. From the 1980s, the participation of stakeholders gained relevance, with its origins in the constructivist paradigm. This stage is what Guba and Lincoln take to be the last generation within the Fourth Generation Evaluation, which would be closer to present-day program evaluation [20]. P é rez Juste states that educational evaluation is the evaluation, based on pre-specified criteria and references, of technically designed and systematically gathered and organised information, regarding the number of relevant factors, which integrate educational processes in order to facilitate the taking of decisions for improvement [33]. 1.4. The Present Study This study is based on the new perspective presented by Escudero regarding evaluation research in the 21st century [ 34 ]. Evaluation is situated in a sphere of social change which aims to resolve problems in real contexts. Mixed methods are used, and the agents involved in the evaluation acquire prominence. Specific justification is made use of for the realities of the object being evaluated. Taking this new perspective of program evaluation as a reference point, our research is based on a real context in which specific needs have been identified and are sought to be resolved. The agents involved gain relevance in the evaluation of the program itself, as their evaluations are gathered on the methodology employed (in the case of the pupils), or of the program itself (in the case of the teachers). This research deals with a new dimension within heritage education research, which conceives program evaluation as a cyclical process that includes planning, action, and evaluation (Table 1). Educ. Sci. 2020,10, 176 4 of 19 Table 1. Outline of the planning, design, and evaluation of the program. Needs assessment (Detection, prioritisation and selection of primary needs) Planning Contact with schools Meta-evaluation Meeting with participating teachers Design of the program based on needs Initial evaluation Coherence and relevance of the program Tools Implementation (one course September-June) Infant education Primary education Secondary education Formative evaluation Assessment of student knowledge Evaluation of partial achievement of the program by teachers and families Changes, adjustments and improvements to the program Summative evaluation Assessment of student knowledge Evaluation of the final results and effects of the program by teachers, families and local authorities It is a new contribution to this field of study based on a review of the model created by P é rez Juste, who proposes integral, integrated and integrating evaluation [ 33 ], taking as a starting point an evaluation of needs in order to formulate objectives which contribute towards resolving the shortcomings or needs found. The program is designed in accordance with the objectives and the initial evaluation (of the design and tools) is carried out, which makes it possible to verify the intrinsic quality of the program (content, technical quality) and its suitability to the context. The next step is the implementation of the program, along with the formative evaluation (the analysis of the fulfilment of the planning, the suitability and effectiveness of the activities and the degree of satisfaction of those involved). There then follows the summative evaluation, which makes it possible to discover the degree of achievement obtained according to the learning outcomes and the impact of the program in the family or the surrounding area. Finally, the meta-evaluation of the whole process, that is, the evaluation of the evaluation itself, is carried out. Introduced by Scriven and developed in depth by Stufflebeam, this is considered to be the last step of program evaluation and makes it possible to take decisions in terms of improvement [35,36]. This paper presents the design and implementation procedure of the program and its initial evaluation, both of the design and of the tools and the formative and summative evaluation as far as the methodology used by the pupils is concerned. This study aims to provide answers to the following questions: • An evaluation of the design of a heritage education program based on the use of ICT and local heritage. Is the design of the program coherent and relevant? • The effectiveness of the use of ICT in the development of heritage education programs. What are students’ assessments of the methodology used? 2. Materials and Methods We present an evaluative model of programs based on the design and implementation of the specifically created program Patrimonializarte. Based on Stake’s premises, a collective study of cases by way of research-action has been designed in which the evaluator constitutes an intrinsic part of the program [ 37 ]. The study is supported by mixed methods, which are commonly used in program evaluation. As defended by Cook and Reichardt, the principle of methodological complementarity has been chosen, with the aim of creating a critical interpretative setting and catering for the evaluative complexity of the program (the diversity of objectives, of agents involved, and recipients of the Educ. Sci. 2020,10, 176 5 of 19 program) [ 38 ]. Particular importance is given to the participation of the agents involved in the program in their own evaluation, as defended by authors such as Tejedor and Fetterman, Kaftarian and Wandersman [ 39 , 40 ]. On the basis of these theoretical considerations, the education program and its evaluation have been designed in accordance with P é rez Juste’s model, which defends an evaluation that is integral (providing for conceptual, procedural, and behavioural objectives), integrated (with the educational activity), and integrating (constituting an active medium, which involves teachers, pupils, and the context in which the evaluation is carried out), always with the ultimate aim of improvement [33]. In accordance with this, once the needs of the teachers had been detected and, based on the comments of a discussion group organised with the participating teachers before the implementation, the program was designed in two phases. Phase 1 focused on activities to raise awareness of heritage for its symbolic appropriation. Phase 2 included activities regarding dissemination and enhancement, which allow the pupils to play an active role in the safeguarding of heritage. In order to evaluate the pupils’ learning, a series of different tools were designed, depending on the pupils’ ages, ranging from interviews, graphic projective tests for infant education (IE), open and closed response questionnaires, and reflective texts to the recording of videos in primary (PE) and secondary (SE) education. At the same time, the evaluation of the program was planned based on different moments: The initial, formative and summative evaluation, in addition to the meta-evaluation of the whole process. In order to achieve this, tools for the evaluation of the program and its impact were designed with the aim of collecting data from the teachers, families and local agents. Both parts (the design of the program and the design of the evaluation) were subjected to the evaluation of external experts prior to being put into action in order to ensure the quality of the program. This is presented in detail in the following sections. 2.1. Context and Participants The design has contemplated the application of the program in the three levels of compulsory education (infant education (IE), primary education (PE), and secondary education (SE)) in two different schools with the aim of identifying dissonances and affinities between different realities (Table 2). We have used an intentional sample chosen according to criteria of viability, practicality, and profitability of the program. The first school (School 1) is located in a rural area of the interior of the province of A Coruña, which has approximately 3000 inhabitants. The second (School 2) is located in a town in close proximity to the city of A Coruña, with a population of approximately 30,000. Two groups can be identified among the participants: a) Agents involved in the education process: Pupils, N =111 (in the first phase) and N =86 (in the second phase, due to the fact that 25 participants from School 2, corresponding to the 3rd year of primary, were lost) and teachers, N =12; b) agents outside of the program: Families, N=43, and local authorities, N =2, the distribution of which can be observed in Table 2. The teachers taught different subjects (physical education, information technology and communication, technology, Galician language and literature, geography, and history). Participation in the program was encouraged from different knowledge areas in order to obtain an interdisciplinary perspective of heritage education. Educ. Sci. 2020,10, 176 6 of 19 Table 2. Participants in the Patrimonializarte program. Agents Directly Involved in the Education Process External Agents Total of pupils per level Total of teachers per level Year and level Pupils Teachers Infants Primary Secondary Infants Primary Secondary Fathers, mothers, legal guardians 22 Local authorities 2 School 1 6th infants 15 1 39 60 12 2 37 5th primary 10 1 2nd secondary 12 7 School 2 6th infants 24 1 21 2 2nd primary 25 1 3rd primary 25 1 Educ. Sci. 2020,10, 176 7 of 19 2.2. Procedure Prior to Implementation An initial informative meeting regarding the program was held at the schools with the aim of involving the participating teachers. Once the number of participants had been reached, a discussion group was organised in which the implementation process, with the consensus of the teachers, was structured in the following way: • Information and planning: The teachers inform the pupils of the planning and methodology which will be carried out throughout the academic year. • Performance of classroom tasks: The pupils carry out the planned activities in order to achieve the aims proposed in the program. • Carrying out of joint tasks: Activities in which the three levels converge (transmitting results from one year group to another, school celebrations, exhibitions open to the school and local community). • Evaluation: Evaluation of the knowledge acquired by the pupils and of the program by both pupils and teachers. • Knowledge transfer: The final products of the program are presented to both the school and the local community via digital resources. A second discussion group was held with the teachers and different organisational roles were agreed upon (Table 3). Table 3. Organisational chart. Roles of the Agents Involved in the Program Teaching staff To review the content guide, activities, competences, teaching aims, criteria and standards of evaluation provided by the external evaluator in order to propose changes and improvements. To establish the number of work sessions and the annual calendar, to review the tasks and grouping of pupils for the work of the program. To monitor the degree to which the objectives of their part of the program are achieved. To guide the pupils towards independent learning, motivating them to work autonomously and offering all the information and resources needed during the implementation of the program. To evaluate the program according to the criteria and standards set out in the evaluation tools. Coordinator of the program in the school To call the participating teaching staffto the preparatory and monitoring meetings of the program and advise of any possible changes in the schedule. To collect the evaluation tools from the teachers. To resolve any doubts which may arise during the course of the program in collaboration with the external evaluator. External evaluator To draw up a guide to the program, with contents, activities, competences, teaching aims, criteria and standards of evaluation for each year group. To carry out participant observation during the implementation in order to check the evolution of the program, its problems and possible improvements. To provide all necessary help at each stage of the application of the education program and evaluate the entire process. Educ. Sci. 2020,10, 176 8 of 19 2.3. The Design of the Program A perspective of integral, integrated, and inclusive evaluation was assumed, based on the new lines of action in education research [ 33 , 34 ]. Education intervention is understood within a process that combines the planning, execution, and evaluation of the program in parallel. Bearing this in mind, the research is structured according to different moments: • The evaluation of necessities (detection, prioritisation, and selection of primary needs and the design of the program based on these aspects); •The initial evaluation (of the design and the tools); •The implementation of the program; •The formative evaluation (changes, adjustments and improvements to the program); •The summative evaluation (analysis of the results and effects of the program); •Meta-evaluation. In accordance with this outline, the objectives of the program have been established and the design has been adapted to the diagnosis of necessities: • To design activities relating to heritage elements in proximity to the students in order to stimulate their motivation and identification. • To include activities that contemplate the use of digital resources in order to encourage the pupils’ motivation. The program is structured in two phases (Table 4). Table 4. The distribution of the program and monitoring of evaluation. Phase 1 Phase 2 Agents Objectives Methodology Agents Objectives Methodology Students Know Respect Preserve Understand Value Enjoy Activities to raise awareness and processes of identification Opening up to the family and education community Students Transmit Act Disseminate Activities for taking action regarding heritage and dissemination General opening up Teachers Teachers Family Family Education community Education community Local community Heritage managers Evaluation of needs and initial evaluation Formative evaluation Summative evaluation and meta-evaluation Taking local heritage as a starting point, it is related to the territory in which it is located in order to achieve a greater level of awareness and feeling of identification on the part of the pupils [ 41 ]. The objective is for them to recognise their everyday ordinary space in order to make it easier for them to understand the heritage in an integrated and contextualised way. Activities employing technology (e-learning and m-learning) are included in order to increase the students’ motivation (Table 5). In the evaluation of their needs, the teachers considered that innovative methodologies were those that best captured the attention of the children. New technologies are useful tools for the study of heritage and form the basis for introducing new methodologies, thus leading to digital literacy [18]. Educ. Sci. 2020,10, 176 9 of 19 Table 5. Information and Comunication Tecnology (ICT) activities. Activities Phase 1 Activities Phase 2 Infant Education (IE) A1 Classroom work with digitalised old photographs: School, the passing of time No activities were carried out relating to ICT Primary Education (PE) A2 Classroom work with digitalised old photographs: The local area and traditional professions A6 Field trip to record audio-visual material: forgotten heritage A3 Field trip to work with photography: Perspectives and shots, changes and continuities in the local area A7 Editing of audio-visual materials for dissemination with the CyberLink PowerDirector program Secondary Education (SE) A4 Classroom work with digitalised old photographs: Changes and continuities in the local area A8 Work with the Geoaumentaty program: geolocation and geo-referencing of heritage elements researched by all the groups A5 Editing of audio-visual materials with the Ciberlink Power Director program: Changes and continuities in the local area A9 Work with the BlocksCAD program and Ultimaker Cura for the editing and design of a 3D dolmen The design of the program incorporates contents, competences, and objectives extracted and adapted from the official curricula of infant, primary, and secondary education in the Autonomous Community of Galicia. The aim of phase 1 was to enable the pupils to identify with their local area by way of working with digitalised old photographs of their surrounding area (school, trades, places). The SE pupils worked with the CyberLink PowerDirector program (Figure 1) to edit and put together a documentary video on changes and continuities in their local area. The work they produced was openly presented to the whole school and included information via QR codes, which redirect to the school’s webpage. Educ. Sci. 2020, 10, x FOR PEER REVIEW 9 of 19 Table 5. Information and Comunication Tecnology (ICT) activities. Activities Phase 1 Activities Phase 2 Infant Education (IE) A1 Classroom work with digitalised old photographs: School, the passing of time No activities were carried out relating to ICT Primary Education (PE) A2 Classroom work with digitalised old photographs: The local area and traditional professions A6 Field trip to record audio-visual material: forgotten heritage A3 Field trip to work with photography: Perspectives and shots, changes and continuities in the local area A7 Editing of audio-visual materials for dissemination with the CyberLink PowerDirector program Secondary Education (SE) A4 Classroom work with digitalised old photographs: Changes and continuities in the local area A8 Work with the Geoaumentaty program: geolocation and georeferencing of heritage elements researched by all the groups A5 Editing of audio-visual materials with the Ciberlink Power Director program: Changes and continuities in the local area A9 Work with the BlocksCAD program and Ultimaker Cura for the editing and design of a 3D dolmen The design of the program incorporates contents, competences, and objectives extracted and adapted from the official curricula of infant, primary, and secondary education in the Autonomous Community of Galicia. The aim of phase 1 was to enable the pupils to identify with their local area by way of working with digitalised old photographs of their surrounding area (school, trades, places). The SE pupils worked with the CyberLink PowerDirector program (Figure 1) to edit and put together a documentary video on changes and continuities in their local area. The work they produced was openly presented to the whole school and included information via QR codes, which redirect to the school’s webpage. Figure 1. Pupils putting together the documentary video using the Cyberlink PowerDirector software. In phase 2, the objective was for the schoolchildren to recognise the forgotten heritage of their local area in order to be able to perform tasks relating to it. Activities were designed to disseminate this heritage and to attempt to turn it into living heritage of the community. In addition to the audiovisual activities, the students were encouraged to create a final product with the geolocation software Geoaumentaty, so that all of the groups could have a point of convergence in the program. The knowledge acquired was transferred to the local community with the aim of building learning networks and promoting service-learning. This geolocation augmented reality (AR) tool makes it possible to see the points of interest (POI) created by the pupils in real time. By using the physical coordinates and the compass on the mobile device, they are able to see, in the actual place, virtual information about each site. This tool has a content management system (Figures 2 and 3) in which the pupils can input, edit, and publish Figure 1. Pupils putting together the documentary video using the Cyberlink PowerDirector software. In phase 2, the objective was for the schoolchildren to recognise the forgotten heritage of their local area in order to be able to perform tasks relating to it. Activities were designed to disseminate this heritage and to attempt to turn it into living heritage of the community. In addition to the audio-visual activities, the students were encouraged to create a final product with the geolocation software Geoaumentaty, so that all of the groups could have a point of convergence in the program. The knowledge acquired was transferred to the local community with the aim of building learning networks and promoting service-learning. This geolocation augmented reality (AR) tool makes it possible to see the points of interest (POI) created by the pupils in real time. By using the physical coordinates and the compass on the mobile device, they are able to see, in the actual place, virtual information about each site. This tool has a content management system (Figures 2and 3) in which the pupils can input, edit, and publish Educ. Sci. 2020,10, 176 16 of 19 motivation of the pupils. They were of the opinion that other types of more traditional or less dynamic activities would not attract the attention of the pupils in the same way. With regard to the results of the summative evaluation, in category C1, the teaching staffevaluated the effectiveness of the program in terms of the acquisition of competences. Ten of the 12 teachers who participated awarded the program a high score in terms of effectiveness, stating that it had contributed towards improving the acquisition of key competences such as digital competence. For category C4, referring to the impact of the program, 9 of the 12 positively evaluated the repercussion of the program in the press, exhibitions, and the digital media, making mention of its benefits in the dissemination of the heritage in question. The remaining three teachers did not provide an answer to this item. 4. Discussion and Conclusions The main objectives of this study were, first of all, to evaluate the coherence and documental composition of the design of the Patrimonializarte program and the relevance and clarity of the tools used based on the judgement of experts. Secondly, it aimed to evaluate the effectiveness and usefulness of the digital resources used in the program based on the judgement of the agents involved (pupils and teaching staff). 4.1. Program Design Following the analysis of the results of the evaluation of the design of the program and the calculation of the Bangdiwala’s weighted agreement coefficients [ 43 ], an optimal degree of agreement was shown in terms of its coherence and documentary composition. In the variable “documental composition”, a weighted agreement coefficient B WN =0.883 was obtained, slightly less than that of the variable “coherence of the design”. Although this result is excellent, according to the criteria of Muñoz and Bangdiwala [ 44 ], its slight decrease is due to the fact that the judges considered that the chronological planning should have been more specific. However, the nature of the fieldwork itself made it necessary to carry out a more general planning given that it should be adapted to the temporal realities of the schools [ 34 ]. Following the same criteria, the Bangdiwala’s weighted agreement coefficients demonstrate excellence in the relevance and clarity of the tools employed in the program. A program that is solid and structured in terms of its documental composition, its organisation, and the monitoring of its evaluation has been designed. Thus, the tools designed for the evaluation also had an extremely high degree of agreement. For this reason, they can be considered to be relevant and coherent in relation to the program. However, the diversity of the tools employed constitutes both a strength and a weakness at the same time. As a collateral effect, the systematisation and later analysis of the information proves to be complex. On the other hand, its strength lies in the fact that this variety of tools makes it possible to measure the results of the application of the program in the different groups in a more individualised manner. 4.2. Digital Resources The results extracted from the evaluations of the digital resources in the program by the pupils and teaching staffindicate that there are no significant differences between the two agents, given that both consider that a methodology supported by ICT tools for working with heritage is positive. One strength in common lies in the variable “dissemination”, as the results from Phase 2 (categories C4 and C5) show that the pupils lend importance to the activities carried out largely thanks to the fact that they enable them to disseminate their work. This may be due to the fact that, via working with ICT tools, the pupils themselves become the creators of the contents and the protagonists of their own learning, thereby motivating them to carry out the activities [ 45 ]. In this regard, the teaching staffremarked that a positive point of the use of ICT tools was the motivation that this methodology offers to the pupils and that, via their use, the pupils acquired competences, among them digital competence [11]. Educ. Sci. 2020,10, 176 17 of 19 As far as category C4 is concerned, relating to the significant learning acquired by PE pupils, there is a certain degree of incongruence in the results. When the schoolchildren were asked what discoveries they had made through their work with digital photographs, in PE, 46.6% stated that they had identified certain changes and continuities which had taken place in their local area. In SE, this figure was 63.6%. The majority of PE pupils stated that working with digital photographs had enabled them to learn about the past, in addition to being able to identify changes and continuities. However, only 27.3% in SE mentioned that this activity had helped them to learn about the past. The SE pupils stated that they had noticed changes and continuities thanks to the activities, but they claimed to have obtained a lesser degree of knowledge about the past. This may be due to the typology and methodology used in the different activities. In PE, a field trip was organised, in which the pupils (A3) could verify in situ the changes and continuities that had taken place in their local area via the taking of photographs from the same perspective as that of the old photographs. In line with this, the PE pupils positively evaluated the field trip for the work with digital resources in category C5. This activity may be more effective than the activity carried out in SE (A5), in which an audio-visual document was created counterposing old photographs and the songs of a local author, which relate what life there was like in the past. Perhaps this activity requires the pupils to think with a higher level of abstraction in order to visualise changes and continuities. Thus, they did not obtain so much knowledge of the past as their counterparts in PE. However, 83.3% of the pupils considered that the making of audio-visual documents was relevant, with the justification, again, of the importance of dissemination. 4.3. Main Conclusions Many studies on the use of new technologies in heritage education and on the evaluation of heritage education programs have already been carried out. However, there is a shortcoming in this area that must be remedied; an integrated, integral, and integrating evaluation of the programs [ 33 ]. For this reason, a design has been specifically created, which has been evaluated and implemented in order to then be able to evaluate its results in an active manner. The study has achieved its proposed objectives. The reliability of the program and of its evaluation tools has been measured and proven and the evaluations made by the pupils and teachers regarding the use of new technologies in heritage education have been analysed. The results show that employing elements of heritage that lie in the learners’ immediate proximity generates a high level of interest. E-learning and m-learning are motivating methodologies that allow pupils to acquire significant knowledge by learning to do things. The pupils give importance to the use of ICT tools due to their great potential in the dissemination of heritage. The teachers believe that their students feel more involved in a methodology in which they have a leading role and generate contents themselves, seeing their work reflected in a webpage or a mobile application. One limitation detected is that the design of activity A5 for SE did not fully achieve the proposed objectives. The pupils have a better understanding of the passing of time and acquire more knowledge about the past by seeing the changes which have taken place in situ, rather than doing work in the classroom, as has been proven in PE. This study is a contribution to program evaluation research in heritage education, given that it deals with evaluation as a cycle that encompasses the time before, during, and after its implementation. The usefulness of ICT tools is corroborated in the study of heritage. We believe that future lines of research should approach evaluation at all moments of the program, not only at its end, in order to respond to the demands of the agents involved, to make it possible to carry out improvements or adjustments to the program and to emit value judgements in order to improve heritage education. In this way, contributions can be made towards improvements in the field of heritage education. Author Contributions: Conceptualization, L. C.-C.; Data curation, J. R.-M.; Formal analysis, L.C.-C. and J. R.-M.; Funding acquisition, R.L.-F.; Investigation, L.C.-C.; Methodology, L.C.-C.; Project administration, R.L.-F.; Resources, Educ. Sci. 2020,10, 176 18 of 19 L.C.-C.; Software, J.R.-M.; Supervision, C.J.G.-C. and R.L.-F.; Validation, C.J.G.-C. and R.L.-F.; Writing—original draft, L.C.-C.; Writing—review & editing, L.C.-C., J.R.-M., C.J.G.-C. and R.L.-F. All authors have read and agreed to the published version of the manuscript. Funding: This article has been posible thanks to the research projects “Emotional dimensions and controversial issues in teacher education (PGC2018-094491-B-C31) & “Methodological concepts and active learning methods for the improvement of teaching competencies” (PGC2018-094491-B-C33), funded by The Ministry of Science, University and Innovation, co-funded by FEDER, and project “Teacher competencies and active learning methods. An evaluative research with trainee teachers of social sciences” (20638/JLI/18), funded by Seneca Foundation. Agency of Science and Technology from Region of Murcia. Conflicts of Interest: The authors declare no conflict of interest. References 1. Castro-Calviño, L.; L ó pez-Facal, R. Heritage education: Felt needs of the teachers of pre-school, primary and secondary education. RIFOP 2019,94, 97–114. 2. Fontal, O.; Ib á ñez-Etxeberr í a, A. Estrategias e instrumentos para la educaci ó n patrimonial en España. Educ. Siglo XXI 2015,33, 15–32. [CrossRef] 3. Fontal, O. Educaci ó n patrimonial: Retrospectiva y prospectivas para la pr ó xima d é cada. Estud. Pedag ó g. 2016,42, 415–436. [CrossRef] 4. G ó mez-Redondo, C.; Masachs, R.; Fontal, O. Diseño de un instrumento de an á lisis para recursos did á cticos patrimoniales. Cadmo 2017,25, 63–80. [CrossRef] 5. Fontal, O.; Martínez, M. Evaluación de programas educativos sobre Patrimonio Cultural Inmaterial. Estud. Pedagóg. 2017,43, 69–89. [CrossRef] 6. Mar í n-Cepeda, S.; Garc í a-Ceballos, S.; Vicent, N.; Gillate, I.; G ó mez-Redondo, C. Educaci ó n patrimonial inclusiva en OEPE: Un estudio prospectivo. Rev. Educ. 2017,375, 110–135. 7. S á nchez, I.; Fontal, O.; Rodr í guez, J. The Assessment: A pending subject in Heritage Education. RIFOP 2019 , 94, 163–186. 8. Fontal, O.; Garc í a, S.; Arias, B.; Arias, V. Assessing the Quality of Heritage Education Programs: Construction and Calibration of the Q-Edutage Scale. Rev. Psicodidáct. 2019,24, 31–38. 9. Fontal, O.; Garc í a, S. Assessment of Heritage Education programs: Quality standards. ENSAYOS 2019 ,34, 1–15. 10. Ib á ñez-Etxeberria, A.; Kortabitarte, A.; De Castro, P.; Gillate, I. Digital competence using heritage theme apps in the DigComp framework. REIFOP 2019,22, 13–27. 11. Fombona, J.; V á zquez-Cano, E. Posibilidades de utilizaci ó n de la Geolocalizaci ó n y Realidad Aumentada en el ámbito educativo. Educ. XX1 2017,20, 319–342. 12. Ott, M.; Pozzi, F. Towards a new era for cultural heritage education: Discussing the role of ICT. Comput. Hum. Behav. 2010,27, 1365–1371. [CrossRef] 13. Fontal, O.; Garc í a, S. Las plataformas 2.0 como herramientas de aprendizaje y adquisici ó n de competencias en educación patrimonial. RIFOP 2019,94, 285–306. 14. Vicent, N.; Ib á ñez-Etxeberria, A.; Asensio, M. Evaluation of heritage education technology-based programs. VAR 2015,6, 1989–9947. [CrossRef] 15. Calaf, R.; San Fabi á n, J.L.; Guti é rrez, S. Evaluating educational programs in museums: A new perspective. Bordón2017,69, 45–65. 16. Kortabitarte, A.; Ib á ñez-Etxeberria, A.; Luna, U.; Vicent, N.; Gillate, I.; Molero, B.; Kintana, J. Dimensions for learning assessment on APPs regarding heritage. Pulso 2017,40, 17–33. 17. Poce, A.; Agrusti, F.; Re, M.R. Mooc design and heritage education. Developing soft and work-based skills in higher education students. J. E Learn. Knowl. Soc. 2017,13, 97–107. 18. Ib á ñez-Etxeberr í a, A.; Fontal, O.; Rivero, P. Heritage education and ICT in Spain: Regulatory framework, structuring variables and main points of reference. ARBOR 2018,194, a448. 19. Lukas, J.F.; Santiago, K. Evaluación Educative; Alianza Editorial: Madrid, Spain, 2004. 20. Guba, E.G.; Lincoln, Y.S. Fourth Generation Evaluation; Sage Publications: Thousand Oaks, CA, USA, 1989. 21. Tyler, R.W. Changing concepts of educational evaluation. In Perspectives of Curriculum Evaluation; AERA Monograph Series Curriculum Evaluation: Chicago, IL, USA, 1967. Educ. Sci. 2020,10, 176 19 of 19 22. Escudero, T. La Construcci ó n de la Investigaci ó n Evaluativa. El Aporte Desde la Educaci ó n, 1st ed.; Prensas Universitarias de Zaragoza: Zaragoza, Spain, 2011. 23. Cronbach, L.J. Course improvement through evaluation. Teach. Coll. Rec. 1963,64, 672–683. 24. Scriven, M. The methodology of evaluation. In Perspectives of Curriculum Evaluation; AERA Monograph Series Curriculum Evaluation: Chicago, IL, USA, 1967. 25. Stufflebeam, D.L. Educational Evaluation and Decision-Making; Evaluation Center, Ohio State University: Columbus, OH, USA, 1970. 26. Alkin, M. An approach to evaluation theory development. Stud. Educ. Eval. 1979,5, 125–127. [CrossRef] 27. Scriven, M. Goal-free evaluation. In School Evaluation: The Politics and Process; House, E.R., Ed.; McCutchan: Berkeley, CA, USA, 1973; pp. 319–328. 28. Stake, R.E. Evaluating the Arts in Education: A Responsive Approach; Merrill: Columbus, OH, USA, 1975. 29. McDonald, B. Evaluation and the control of education. In Curriculum Evaluation Today: Trends and Implications; Tawney, D., Ed.; McMillan: London, UK, 1976; pp. 125–136. 30. Parlett, M.; Hamilton, D. Evaluation as illumination: A new approach to the study of innovative programmes. In Beyond the Numbers Game; Hamilton, D., Ed.; MacMillan: London, UK, 1977. 31. Eisner, E.W. The Art of Educational Evaluation; The Falmer Press: London, UK, 1985. 32. Joint Committee on Standards for Educational Evaluation. Standards for Evaluations of Educational Programs, Projects, and Materials, 1st ed.; McGraw-Hill: New York, NY, USA, 1988. 33. Pérez, R. Evaluación de Programas Educativos; Muralla: Madrid, Spain, 2006. 34. Escudero, T. La investigaci ó n evaluativa en el Siglo XXI: Un instrumento para el desarrollo educativo y social cada vez más relevante. RELIEVE 2016,22, 1–20. [CrossRef] 35. Scriven, M. An introduction to the meta-evaluation. Educ. Prod. Rep. 1969,2, 36–38. 36. Stufflebeam, D.L. The metaevaluation imperative. Am. J. Eval. 2001,22, 183–209. [CrossRef] 37. Stake, R.E. The Art of Case Study Research; Sage Publications: London, UK, 1995. 38. Cook, T.D.; Reichardt, C.H.S. Qualitative and Quantitative Methods in Evaluation Research; Sage Publications: London, UK, 1982. 39. Tejedor, J. El diseño y los diseños en la evaluación de programas. RIE 2000,18, 319–340. 40. Fetterman, D.M.; Kaftarian, S.J.; Wandersman, A. Empowerment Evaluation: Knowledge and Tools for Self-Assessment, Evaluation Capacity Building, and Accountability, 2nd ed.; Sage Publications: Thousand Oaks, CA, USA, 2015. 41. Pinto, H.; Zarbato, J. Construyendo un aprendizaje significativo a trav é s del patrimonio local: Pr á cticas de Educación patrimonial en Portugal y Brasil. Estud. Pedagóg. 2017,43, 203–227. [CrossRef] 42. Saor í n, J.; Meier, C.; De la Torre-Cantero, J.; Carbonell-Carrera, C.; Meli á n-D í az, D.; Bonnet de Le ó n, A. Competencia Digital: Uso y manejo de modelos 3D tridimensionales digitales e impresos en 3D. EDMETIC 2017,6, 27–46. [CrossRef] 43. Bangdiwala, K. Using SAS software graphical procedures for the observer agreement chart. In Proceedings of the SAS Users Group International Conference, Dallas, TX, USA, 8–11 February 1987; pp. 1083–1088. 44. Muñoz, S.R.; Bangdiwala, S.I. Interpretation of kappa and B statistics measures of agreement. J. Appl. Stat. 1997,24, 105–111. [CrossRef] 45. Fombona, J.; de Goulao, M.F.; Garc í a, M. Melhorar a atratividade da informaç ã o a trav é s do uso da Realidade. Perspect. Cienc. Inform. 2014,19, 37–50. [CrossRef] © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).