T-MOOC, cognitive load and performance: analysis of an experience

Contacto: Antonio Palacios-Rodríguez, [email protected], C. Pirotecnia, s/n, 41013 Sevilla. Proyecto I+D+i “Diseño, Producción y Evaluación de T-Mooc para la Adquisición por los Docentes de Competencias Digitales Docentes”, financiado por el Ministerio de Ciencia e Innovación (RTI2018- 097214-B-C31). http://revistas.um.es/reifop Cabero-Almenara, J., Barroso-Osuna, J. Gutiérrez-Castillo, J.J. & Palacios-Rodríguez, A. (2023). T-MOOC, cognitive load and performance: analysis of an experience. Revista Electrónica Interuniversitaria de Formación del Profesorado, 26(1), 99-113. DOI: https://doi.org/10.6018/reifop.542121 T-MOOC, cognitive load and performance: analysis of an experience Julio Cabero-Almenara, Julio Barroso-Osuna, Juan Jesús Gutiérrez-Castillo y Antonio Palacios-Rodríguez University of Seville Abstract The study of cognitive load allows us to investigate the effectiveness of any training proposal mediated by technology. This paper presents the results of the implementation of a t-MOOC produced following the DigCompEdu Framework of the European Union. The participants are a group of students (n= 148) from the first year of the Pedagogy Degree (University of Seville) of the Educational Technology subject. To do this, the level of cognitive load invested in the interaction with the t-MOOC is analyzed using a validated scale. Secondly, the relationship between the invested cognitive load and the performance achieved in the experience carried out with the contents of two competence areas is studied. After the different analyzes applied, the study concludes that the t-MOOC produced is considered appropriate for the development of digital skills in students. In addition, although the correlations between cognitive load and academic performance were not very high, both are related. In this sense, the potential of training proposals focused on the development of digital skills and the benefits of applying cognitive load studies are discussed. Key words Cognitive load; DigCompEdu; t-MOOC; Higher Education. Fecha de recepción: 29 de septiembre de 2022 Fecha de aceptación: 8 de noviembre de 2022 T-MOOC, cognitive load and performance: analysis of an experience 100 Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) T-MOOC, carga cognitiva y rendimiento: análisis de una experiencia Resumen El estudio de la carga cognitiva permite indagar en la efectividad de cualquier propuesta formativa mediada por tecnología. En este trabajo se presentan los resultados de la puesta en marcha de un t-MOOC producido siguiendo el Marco DigCompEdu de la Unión Europea. Los participantes son un grupo de estudiantes (n= 148) del primer curso del Grado de Pedagogía (Universidad de Sevilla) de la materia Tecnología Educativa. Para ello, se analiza el nivel de carga cognitiva invertido en la interacción con el t-MOOC mediante una escala validada. En segundo lugar, se estudia la relación entre la carga cognitiva invertida y el rendimiento alcanzado en la experiencia realizada con los contenidos de dos áreas competenciales. Tras los diferentes análisis aplicados, el estudio concluye que el t-MOOC producido es considerado como apropiado para el desarrollo de competencias digitales de los estudiantes. Además, aunque las correlaciones entre carga cognitiva y rendimiento académico no fueron muy elevadas, ambas se encuentran relacionadas. En este sentido, se discuten las potencialidades de las propuestas formativas enfocadas al desarrollo de competencias digitales y los beneficios de aplicar estudios de carga cognitiva. Palabras clave Carga cognitiva; DigCompEdu; t-MOOC; Educación Superior. Introduction Cognitive load theory was initially formulated by Sweller (1994). Together with the Multimedia Learning Theory formulated by Mayer (2003), both pursue, as Andrade-Lotero (2012, p.77) points out: “align the design of instructional material with Human Cognitive Architecture (ACH). Its premises are that learners have a very limited working memory capacity when dealing with new information. Therefore, learning will be threatened if instructional materials overload these resources.” This theory is becoming increasingly important in the design and evaluation of instruction, both traditional and technological (Brunke et al., 2003). In short, what it does is apply what is known about human cognitive architecture to the study of learning and instruction to generate knowledge about the characteristics and conditions of effective instruction and learning (Sepp et al. 2019), whether this learning is direct or mediated through technology. As the Center for Education Statistics and Evaluation (2017, p.2) points out: "cognitive load offers theoretical and empirical support to explicit models of teaching or instruction. Research in cognitive load theory demonstrates that instructional techniques are most effective when designed according to how the human brain learns and uses knowledge." In a deeper way, Paas (1992, p.429) defines it as "... a multidimensional concept in which two components can be distinguished, mental load and mental effort. Mental load is imposed by instruction parameters (for example , task structure, sequence of information) and mental effort refers to the amount of capacity that is allocated to instructional demands Manipulations to increase mental load through changes in instruction will only be effective if subjects they are motivated and actually invest mental effort in them. In this study the motivational state of the subjects is expected to be high and equally divided between the Julio Cabero-Almenara, Julio Barroso-Osuna, Juan Jesús Gutiérrez-Castillo y Antonio Palacios-Rodríguez Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) 101 experimental conditions. Therefore, the intensity of the effort is considered an index of cognitive load" . In short, how the subject processes information and how it stores it in shortterm memory, or working memory, and long-term. So "that if working memory is overloaded, there is a greater risk that what is being taught will not be understood by the student, or will be misunderstood or confused, apart from not being effectively stored in long-term memory, so learning will be slowed down” (Centre for Education Statistics and Evaluation, 2017, p.3). Now, what is meant by cognitive load? One of the most specific definitions of cognitive load is the one elaborated by Paas et al. (2003, p.64), who point out that "cognitive load can be defined as the load that the performance of a particular task imposes on the learner's cognitive system." Also synthetically, Kirschner et al. (2010, p.102) indicate that it is “the amount of mental effort that a person spends”. Paas et al. (2003) also point out that it is a multidimensional concept, which implies the relationship of three variables: the characteristics of the task and the learner and an evaluation dimension that reflects the measurable concepts of mental load, mental effort and performance. “Task characteristics that have been identified in CLT research are task format, task complexity, use of multimedia, time pressure, and instructional pacing. Relevant learner characteristics include experience level, age, and spatial ability. Some interactions that have been found are related to age and task format, indicating that instructions involving goal-specific or goal-free tasks disproportionately compromise or enhance older people's learning and transfer performance, respectively" (Paas et al., 2003, p.64). Being the mental load, the cognitive load that occurs in the interaction between the characteristics of the person and the specificity of the task. Although , as indicated by Plass and Kalyuga ( 2019) is also influenced by the emotion aroused by the performance of the task by the person (Plass and Kalyuga, 2019) and by how motivated they are to perform it (Feldon et al., 2019). For their part , Lin and Kao (2018) point out that “cognitive load is affected by causal and evaluation factors. Causal factors include learners' cognitive abilities, complexity of learning tasks, and learning contexts. The interaction of these causal factors induces cognitive load” (p.64). Different authors point out that there are different types of cognitive load. Thus, Leppink et al. (2013) point out 3 types: intrinsic load (IL), external load (EL) and relevant load (GL). Intrinsic load relates to the difficulty of the task or material that the person is intended to learn. There are two factors that condition it: the complexity of the content to be learned and the previous knowledge that the student has of it. On the other hand, the external load is related to the content or the task that is taught and with the instructional characteristics that are not beneficial for learning. Finally, the relevant cognitive load refers to the load imposed on working memory by the learning process itself. That is, by the process of transferring information from long-term memory through the construction of schemes. Therefore, it refers to the characteristics that are beneficial for learning. For their part, Castro-Meneses et al. (2020, p.182) tell us that cognitive load has three types of components: “the intrinsic load (the intrinsic nature and structure of the information and how it interacts with the individual), the external load (the way in which information is presented) and the related load (the cognitive activity required for learning from the stimuli to take place). Regarding its measurement, which is one of the most controversial aspects of the theory (Andrade-Lotero, 2012), it should be noted that we find objective and subjective methods (Brünken et al., 2003; Sweller et al., 2010; Korbach et al., 2017; Castro-Meneses et al., 2020). In the first, there are measures such as the number of errors made by the person, the performance of double tasks, the time taken to perform and execute the task, physiological measurements or electroencephalogram; and in the second self-reports of the T-MOOC, cognitive load and performance: analysis of an experience 102 Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) person on the mental effort invested. Although, as some authors suggest (Naismith et al., 2015), the use of mixed methods is better. As far as self-reports are concerned, a variety of them have emerged: NASA-TLX ( Díaz et al., 2010), the SWAT (Subjective Workload Assessment Technique) scale (Reid and Nygren, 1988), the ESCAM (Subjective Workload Assessment Technique) of Mental Work Load) (Rolo, Díaz and Hernández, 2009 ), or the scale elaborated by Leppink et al., 2013). Scales that share a series of dimensions and as they point out (Rolo et al., 2009, p.36) when validating the ESCAM scale: "The factors obtained in this study are in line with those included in other instruments for assessing the load mental, such as NASA-TLX and SWAT. The ESCAM shares with these instruments the dimensions related to mental and temporal demands. However, while NASA-TLX and SWAT assess some specific consequences of mental workload (eg, performance, frustration, and stress), the ESCAM includes a more general health consequence factor focused on mental fatigue (eg, perception of exhaustion and tiredness).” As Paas et al. (2003, p.66): "Rating scale techniques are based on the assumption that people are capable of introspecting their cognitive processes and reporting the amount of mental effort expended. Although self-assessments may seem questionable, has shown that people are quite capable of giving a numerical indication of their perceived mental workload". For their part, Korbach et al. (2017), acknowledging the criticisms for objectivity, validity and reliability of the subjective scales, point out that the " The advantage of subjective methods is that ratings provide valid information about individual experience. Furthermore, subjective rating scales are very easy to implement and can be used in different learning contexts with diverse learning content and participant groups" (Korbach et al., 2017, p.519). Already focusing on our field of information and communication technologies, Andrade-Lotero (2012) speaks of the following principles: a) problems with free solutions (a decrease in cognitive load is achieved when the problem does not have a single solution) , b) complete problems (this principle is similar to the previous one; however, it assumes the beginning of the development of expertise in the learner), c) divided attention (two sources of information that do not make sense in themselves and that, in turn, time, they must be mentally integrated to understand the information, they should be presented in an associated way), d) modality (two sources of information that do not make sense in themselves must be mentally integrated), e) redundancy (multiple sources of information that have sense in themselves should be reduced to one), f) imagination (mentally repeat the procedures or concepts without the learning materials with which you have already worked) and g) interactive ity (in order to learn with a complex material where different elements interact, it may be necessary to first learn the elements individually and then the relationships that can be established between them). On the other hand , Hollender et al. (2010) talk about different methods to reduce cognitive load, such as the worked example effect, the divided attention effect, the modality effect, and the redundancy effect. The worked example effect indicates that novices benefit from studying worked examples rather than solving conventional problems, as they can focus on useful problem states and solution steps rather than using inefficient strategies that place high demands on working memory. The divided attention effect suggests that multiple sources of visual information must be presented in an integrated fashion if all sources of information are a prerequisite for comprehension. The modality effect also occurs when multiple sources of information are required for comprehension, tending to present them in a single channel (Cabero-Almenara et al., 2022a, 2022b). And the redundancy effect, which implies that the presentation of multiple sources of information that simply reiterate the same information in a different form should be avoided when one source of information is Julio Cabero-Almenara, Julio Barroso-Osuna, Juan Jesús Gutiérrez-Castillo y Antonio Palacios-Rodríguez Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) 103 sufficient for understanding, since having to integrate the redundant information induces a unnecessary memory load. Finally, it should be noted that cognitive load theory has been used to analyze its effectiveness with respect to different uses and designs of different technologies, such as MOOCs, which are the ones we will focus on in this research (Lin and Kao, 2019). Methods Objectives The objectives pursued by the project are different: a) Know the cognitive load that students invest in the interaction with the t- MOOC developed for the acquisition of digital skills according to the DigCompEdu model. b) To know if the students learn the contents developed in two competence areas of the t-MOOC produced. c) To analyze whether there are relationships between the cognitive load invested and the performance achieved in the experience carried out with the t-MOOC. Characteristics of the t-MOOC The t-MOOC produced is based on the DigCompEdu Framework of the European Union (Cabero-Almenara et al., 2020a; Cabero-Almenara et al., 2020b; Casal Otero, et al., 2021; Torres Barzabal et al., 2022). ; Paz Saavedra et al., 2022). Compared to the different types of MOOCs that exist, the one we carry out shares the principles of the so-called t-MOOC, since it focused on carrying out tasks and activities by the person taking it. The elaborated t-MOOC has the following resources: • 66 learning modules (3 for each DigCompEdu competency: beginner, intermediate and advanced). • 230 tasks distributed in the learning modules. • 1 general forum. • 1 welcome forum. • 6 area specific forums. • 15 task-specific forums. • 6 self-diagnostic tests of the level of competence (beginning, intermediate and advanced). • 1 animation with navigation instructions and use of the t-MOOC. • 1 general animation (DigCompEdu). • 6 specific animations for each DigCompEdu competence area. • 22 animations specific to each DigCompEdu competition. • 16 animations integrated in the different learning modules. T-MOOC, cognitive load and performance: analysis of an experience 104 Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) • 24 infographics integrated into the different learning modules. • 11 polymedia integrated in the different learning modules. The type of tasks used has been of different types: participation in a forum; explain the steps for the development of a product, resource or virtual community; carry out an activity with the padlet program; identification of errors made in a sequence to carry out some type of activity; realization of a conceptual map; making a power-point or prezzi presentation; recording of a simulation of a conference presentation; describe different synchronous and asynchronous communication tools… A wide variety of programs have been used for the production of the TMOOC. Specific: • ExeLearning (learning modules) • VYOND (educational animations) • Genially (Infographics) • Photoshop (graphic design and aesthetics of the TMOOC) • Adobe Premiere (video editing) • Audacity (audio equalization) All the previously mentioned elements that make up the elaborated t-MOOC were hosted on a Moodle platform, following the following operation. First, the students are registered so that they can have a user and specific credentials. After authenticating, the students enter the so-called “main page”, where there is a video tutorial to help in the virtual environment and a generic animation on the DigCompEdu Framework (Figure 1). Figure 1. Animation on the DigCompEdu Framework. Source: self made. The general structure of the TMOOC is as follows. After having viewed the two videos mentioned above, the areas of competence to work on are found (Figure 2). Each competency area has a series of related competencies (Figure 2). For each competition, there is also an explanatory animation of it. Julio Cabero-Almenara, Julio Barroso-Osuna, Juan Jesús Gutiérrez-Castillo y Antonio Palacios-Rodríguez Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) 105 Figure 2. Presentation of Area 1. Source: Own elaboration. In each area, students must complete a diagnostic evaluation (Figure 3) that determines the level of digital competence they have: basic, intermediate or advanced. Depending on the level, students are referred to a specific learning itinerary in order to improve it. Figure 3. Example of item available in the diagnostic evaluation. Source: self made. In short, each competence to work on within each area has three different itineraries (learning modules). Each learning module has an associated content module and task module. The content blocks have explanatory texts, infographics, polymedia, animations, hyperlinks... In the case of the task blocks, students can choose between different types of activities to be able to pass it. The presentation of the activities is carried out through a guide where different aspects are incorporated, such as: their identification, recommendations for their realization, a checklist for the teacher to verify the quality of the delivery and an evaluation rubric that is used by the students. TMOOC tutors. The type of activities is diverse: making concept maps, participation in forums, building a blog, creating PLEs with certain tools, organizing activities for students... Sample The research involved 148 students who were studying the discipline of "Educational Technology" in the Pedagogy Degree taught by the Faculty of Educational Sciences of the University of Seville. The students were volunteers and carried out the experience in a training period that lasted three months. The activities carried out by the students were tutored and corrected by teachers from the research team. T-MOOC, cognitive load and performance: analysis of an experience 106 Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) In the research, the students worked with two DigCompEdu competency areas: “digital resources” and “digital pedagogy”. 91.9% (136) were female and 8.1% (12) male, with a mean age of 20.76 years. Instrument The instruments for collecting information were two: one for the analysis of cognitive load and another for the analysis of learning performance. For the first of them, the scale developed by Leppinke et al. (2014), which extended by three items the one formulated a year earlier by the same authors (Leppink, et al., 2013) to analyze the mental effort invested. The difference between the two instruments is not only found in the number of items, but also that, in addition to measuring the intrinsic load, the external load and the relevant load, it incorporates a new dimension that is "mental effort"- Specifically, the instrument used was the following: • [1] The contents of the t-MOOC were very complex. • [2] The tasks covered by the t-MOOC were very complex. • [3] Very complex terms were mentioned in the t-MOOC. • [4] I invested a great deal of mental effort into the complexity of this activity. • [5] The explanations and instructions of the t-MOOC were very unclear. • [6] The explanations and instructions of the t-MOOC were full of unclear terms. • [7] The explanations and instructions of the t-MOOC were, in terms of learning, very ineffective. • [8] I put a lot of mental effort into understanding the unclear and ineffective explanations and instructions built into the t-MOOC. • [9] This activity really improved my understanding of the content that was developed in the t-MOOC. • [10] This activity really improved my understanding of the problems that were developed in the t-MOOC. • [11] This activity really improved my knowledge of the terms that were mentioned in the t-MOOC. • [12] This activity really improved my knowledge and understanding of how to deal with the issues covered in the t-MOOC. • [13] I invested a lot of mental effort during this activity to improve my knowledge and understanding of the content developed in the t-MOOC. Where items (1 to 3) measure internal cognitive load, 5 to 7 measure external cognitive load, 9 to 12 pertinent cognitive load and items 4-8-13 mental effort. After analyzing the reliability index, both for the instrument as a whole and for the different dimensions that make it up, Table 1 shows the values achieved. Julio Cabero-Almenara, Julio Barroso-Osuna, Juan Jesús Gutiérrez-Castillo y Antonio Palacios-Rodríguez Revista Electrónica Interuniversitaria de Formación del Profesorado (REIFOP) 107 Table 1. Reliability index of the instrument. Dimensions Alpha Internal cognitive load .935 External cognitive load .965 Relevant cognitive load .946 Mental effort .976 Total .975 The results obtained, both in the overall nature of the instrument and in the different dimensions that make it up, indicate, according to Mateo (2004), that the instrument has high reliability indices. On the other hand, for academic performance, a multiple choice test was used regarding the contents that were developed in the two areas of competence. Instruments that were applied in the pretest and posttest modality, and that had a multiple choice type construction. Results In the first place, we will present the means and standard deviations reached in the different items of the instrument (Table 2). Table 2. Mean and standard deviation of the items of the cognitive load instrument. Mean SD [1] The contents of the t-MOOC were very complex. 4.38 2.35 [2] The tasks covered by the t-MOOC were very complex. 4.43 2.34 [3] Very complex terms were mentioned in the t-MOOC. 4.43 2.23 [4] I invested a great deal of mental effort into the complexity of this activity. 5.32 2.60 [5] The explanations and instructions of the t-MOOC were very unclear. 3.57 2.46 [6] The explanations and instructions of the t-MOOC were full of unclear terms. 3.57 2.59 [7] The explanations and instructions of the t-MOOC were, in terms of learning, very ineffective. 2.95 2.06 [8] I put a lot of mental effort into understanding the unclear and ineffective explanations and instructions built into the t-MOOC. 3.84 2.79 [9] This activity really improved my understanding of the content that was developed in the t-MOOC. 7.84 1.64