Cognitive Technology to Evaluate the Academic Learning of Computational Cognition in Psychology Students

Research Article https://doi.org/10.12973/ijem.10.2.1013 International Journal of Educational Methodology Volume 10, Issue 2, 213 – 225. ISSN: 2469-9632 http://www.ijem.com/ Cognitive Technology to Evaluate the Academic Learning of Computational Cognition in Psychology Students Guadalupe Elizabeth MoralesMartinez* National Autonomous University of Mexico, MEXICO Angel Garcia-Collantes Distance University of Madrid, SPAIN Rafael Manuel Lopez-Perez Fundación Universitaria Behavior & Law, SPAIN Received: November 19, 2023 ▪ Revised: January 2, 2024 ▪ Accepted: March 14, 2024 Abstract: This study illustrated an alternative way to evaluate students’ academic learning. It involved the joint and intertwined application of the natural semantic network technique, computer simulations, and semantic priming experiments to assess the cognitive changes in knowledge structures due to academic learning in two groups of psychology students. The experimental group was enrolled in a course on computational cognition, while the control group was oblivious to this course. The results indicated that the cognitive assessment tools discriminate the cognitive changes produced as a result of general training undertaken in a psychology degree versus the influence of a specific course. After the course, the experimental group increased their technical vocabulary, changed their conceptual valuation of definers related to computational theories of mind, and reorganized the relations among definers according to the computational cognition approach. Also, this group presented a higher connectivity index between the concepts of the semantic network, their conceptual activation level and conceptual coactivation pattern changed, and their access level to the evaluated schema’s concepts improved. In contrast, the control group did not show significant changes in their cognitive patterns after the course. These findings suggest that cognitive tools may be helpful in the diagnosis of academic learning. Keywords: Academic learning, cognitive assessment, natural semantic networks, psychology students, semantic priming. To cite this article: Morales-Martinez, G. E., Garcia-Collantes, A., & Lopez-Perez, R. M. (2024). Cognitive technology to evaluate the academic learning of computational cognition in psychology students. International Journal of Educational Methodology, 10(2), 213225. https://doi.org/10.12973/ijem.10.1.1013 Introduction Learning assessment is one of the most exciting challenges of the 21st century education, requiring evolution in the vein of teaching models and technology advancements and in regard to understanding society's emerging needs. In addition, determining aspects of learning assessments and how they should be evaluated in educational settings is a major, very difficult task (National Research Council [NRC], 2001; Nichols & Sugrue, 1999). Most academic tests measure students' short-term memory skills as knowledge acquisition; for example, teachers administer multiplechoice, true-false, or open-question tests at the end of a course to verify whether the student has retained learned knowledge. The correct answer is proof of the student's acquired knowledge, but such conventional evaluation does not determine if a student has developed a long-term cognitive ability; many students use strategies to pass exams without engaging in meaningful long-term learning (Marzano, 1994; Marzano & Costa, 1988; Marzano et al., 1990). A contemporary evaluation of academic learning requires the use of evaluation tools to measure abilities related to cognitive information processing, which are central to training 21st-century students, who live in an economy largely dependent on information management (Arieli-Attali, 2013). However, there are scarce alternatives embracing this modern vision of learning, technology developments, and advancements in science learning. In line with this concern, the NRC (2001) used the advancement of cognitive science to rethink assessment approaches, since advances in cognitive science expanded the knowledge surrounding important learning dimensions. Further, its advances in measurement techniques open possibilities to understand and interpret increasingly complex evidence relating to the academic performance of students. * Corresponding author: Guadalupe Elizabeth Morales-Martinez, National Autonomous University of Mexico (UNAM), Mexico.  [email protected] © 2024 The author(s); licensee IJEM by RAHPSODE LTD, UK. Open Access - This article is distributed under the terms and conditions of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/). 214  MORALES-MARTINEZ ET AL. / Cognitive Technology to Evaluate the Academic Learning In consonance with the NRC’s vision, Morales-Martinez et al. (2023) highlighted the usefulness of cognitive assessment tools to evaluate the process and results of learning, since measurement advances in cognitive psychology are compatible with new technologies. The intertwined use of cognitive paradigms to study the human mind with computational developments offers an opportunity to create or innovate digital methods to approximate the learning assessment, as is illustrated in the next section. Literature Review The proposal of linking cognition and learning theories with learning assessment practices and teaching is not a new concept (see NRC, 2001), Arieli-Attali (2013) noted how since the previous century, there have been sophisticated approaches to enchain cognitive science with learning assessment. More recently, Lopez-Ramirez et al. (2014) proposed the Chronometric Constructive Cognitive Learning Evaluation Model, or C3 LEM, which works under the principles of serial and parallel human information processing and provides cognitive tools that explore how the student's mind works when forming knowledge structures. Conventionally, the C3 LEM suggests using intertwined chronometric and mental representation techniques to assess academic learning processes (selection, storage, and retrieval of the information stored in the students' memory). The application of C3-LEM involved two phases (Figure 1): the constructive cognitive evaluation and the chronometric cognitive evaluation of knowledge. The first assesses the meaning formation on knowledge mental representation and reveals their cognitive characteristics (the conceptual organization, structure, and dynamic) through mental representation techniques (Natural Semantic Networks) and computational simulations. Figure 1. Phases and Components of the C3-LEM (Morales-Martinez, Angeles-Castellanos, et al., 2020) The Natural Semantic Networks (NSN) technique (Figueroa et al., 1976) is a methodological approach to studying mental representation, which implies the recovery of information from human memory by using conceptual clues (targets). This technique is useful for exploring the development of knowledge schemas acquired throughout the academic year. According to Morales Martínez, López Pérez, et al. (2020), applying an initial NSN in a course and comparing it to the final NSN allows for observation of the qualitative (content) and quantitative cognitive changes due to academic learning. Comparison of both NSN, initial and final, involves contrasting several indicators such as the semantic richness (number of conceptual nodes), semantic relevance of the concepts (M value), and the Inter-Response Times (temporary patterns of definers appearance), among others. The NSN technique requires a definition task; participants define core target concepts of the academic course using definers (nouns, verbs, adjectives, pronouns). Following this, they rate these definers based on their relationship degree with the target (Morales-Martinez, Angeles-Castellanos, et al., 2020). These scores support the computational simulations of the knowledge schema's behavior at the beginning and end of the academic year (Lopez-Ramirez et al., 2014). Lopez-Ramirez et al. (2015) used this technique to explore schematic activity in academic learning across different knowledge domains. Furthermore, computer simulations can indicate connections among concepts only observed with these tool types. For example, Gonzalez et al. (2013) observed that high school students established implicit relationships among concepts of an NSN on moral course. They observed the concept of parent co-activated police, even though these concepts appear semantically unrelated. However, when the researchers considered the social and cultural context of the participants, they noted a relationship of psychological significance. In general, the NSN analysis and the computer simulations allow us to observe the changes in the mental representation of students' knowledge produced by learning a knowledge domain. Combining these techniques with experimental cognitive studies increases the potential to evaluate other relevant aspects of academic knowledge as the temporal patterns of schematic behavior. For example, the implementation of chronometric cognitive evaluation studies helps obtain temporal trends in the capacity to access information from memory. International Journal of Educational Methodology  215 Morales-Martínez and Santos-Alcantara (2015), following the proposal of E.-O. López-Ramírez (personal communication, August 9th, 2014), used the Inter-Response Time (IRT) of NSN to analyze the information accessibility level. These authors reported that the definers with the greatest weight tend to appear between 30 and 40 seconds, and they appeared between the third and fifth positions on the list. However, there was no discussion about what this positioning means. It is unknown if this is the case for all knowledge domains and which variables influence the recovery of academic concepts. In addition to IRT, reaction times (RT) are another indicator of the chronometric cognitive evaluation; these come from schematic word recognition studies applied before and after the course. The RT can be classified through a neural network to discriminate whether there was an integration of the academic content into students’ long-term memory structures at the end of a course. To achieve this, the C3-LEM includes the application of the semantic priming paradigm throughout lexical decision tasks (McNamara, 2005), which consists of presenting word pairs with different relationship types (e.g., associative, categorical, schematic, unrelated). The experimental task is to mentally read the first (prime) and the last (target) word and decide whether the target is spelled correctly. The recognition times of targets offer information about how the context preceding schematic words affects students' information processing. Suppose the presentation of a stimulus (word or image) is preceded by another semantically related stimulus. In that case, students will recognize the second stimulus more quickly or accurately than in the case of no semantic relationship between the two stimuli. In a C3-LEM study, the semantic relationships between the schematic word pairs are relevant. If the knowledge schema does not exist in the student's memory at the beginning of the course, and by the school year end, students have integrated the information into their knowledge structures, the recognition time of schematic pairs will significantly reduce by the course end. Lopez (1996) and Lopez and Theios (1992) proposed that the semantic priming effect produced by a schematic relationship is a concept termed “schematic priming.” In academic learning, schematic priming is present just for those word recognition tasks that involve concepts related to the knowledge schema of the evaluated course. The evidence from cognitive studies shows that when a student stores the conceptual nodes learned in class in his long-term memory, the word recognition times related to the learned schema decrease at the course end (e.g., see Gonzalez et al., 2013; Morales Martínez, López Pérez, et al., 2020). The opposite happens when students do not consolidate the information in their memory (Urdiales-Ibarra et al., 2018). In short, this type of knowledge organization cognitive phenomenon in long-term memory (schematic priming) can account for a learning process on academic knowledge schemas. Techniques such as those involved in the C3-LEM can be valuable tools in the assessment for, as, and of learning (see Morales-Martinez, 2020; Morales-Martinez & Lopez-Ramirez, 2016; Morales-Martinez et al., 2015, 2017), thus, it is essential to accumulate evidence of the worth of this cognitive approach to evaluate learning effectively. Therefore, using this evaluation model, this research explored the changes in students' knowledge structures due to the learning conducted in a face-to-face psychology course. The first question in this research was to determine the pattern of cognitive knowledge structure changes that students experience after learning a specific topic (Computational Cognition). The second question was to discriminate the pattern of cognitive changes among students enrolled and not enrolled in the course; as the results of this study pointed out, there are differences in cognitive patterns related to expertise in this field. The cognitive measurement tools accounted for cognitive differences between the change directed by specific learning and the spontaneously produced by non-systematic exposure to general psychology education. Methodology Research Design The present authors used a qualitative and quantitative mixed method (C3-LEM) to measure the cognitive dimension of academic learning in a Computational Cognition course. First, they designed and applied an NSN study; then, they performed a computational simulation on the data of this first study using a neural network of constraint satisfaction proposed by Lopez and Theios (1992). Finally, they implemented experimental research through the semantic priming paradigm. Sample Two groups of first-year psychology students participated in this study (74 women and 30 men). The first attended a computational cognition course (experimental group), and the second was the control group, which did not have access to the information on the subject evaluated. The control and the experimental groups came from different institutions. The participants had a mean age of 19 years (range 17 to 27 years, SD= 1.7). Participation was voluntary, and all participants gave informed consent. 216  MORALES-MARTINEZ ET AL. / Cognitive Technology to Evaluate the Academic Learning Instruments The authors selected ten stimuli from the theory discussed in the course (mind, computation, computational mind, HIP, von Neumann, Turing machine, connectionism, memory, working memory, and long-term memory) to design the NSN study. Concept selection followed Morales-Martinez's Protocol for the Collection of Concepts (Morales-Martinez, 2015), which consists of several conceptual analysis steps on the subject for evaluation. This is a guide to selecting target concepts to create the NSN study as well as 30 definers to design the semantic priming experiment. The authors organized the definers in prime-target pairs (e.g., software-processes, algorithm-language, and memory-processor). Furthermore, they added 15 associative concept pairs (e.g., bee-sting, airplane-pilot, dentist-tooth) and 15 unrelated word pairs (e.g., floor-screen, mountain-blood, war-elevator). Procedure This research involved an announcement, where the students received an invitation to participate voluntarily, also they learned about the study objectives and benefits, and the authors gave a privacy warning about their data. Following this, the students who chose to participate gave their informed consent and received specific instructions for NSN and semantic priming study tasks. Finally, the students completed both tasks, and the authors performed a computer simulation on the data from the first study. Natural Semantic Network Study: The Mental Representation of Knowledge The participants performed a conceptual definition task, which involved defining ten target concepts that embraced the computational cognition topic. The targets appeared randomly, one by one, and remained in the computer screen's center for 60 seconds. The participant defined each target with verbs, nouns, adjectives, and pronouns and later qualified individually the conceptual quality of each definer concerning its target. As Lopez (1996) and Lopez and Theios (1992) suggested, conceptual scores varied between one and ten; the smaller the number, the lower the quality of the schematic relationship between the definer and its target. Computer Simulation: The Schematic Dynamics of Knowledge Structures Data obtained from the NSN study fed a constraint-satisfaction neural network using EVCOG software. The computational simulation procedure followed Lopez and Theios' (1992) formula: WIJ = -1n{[p(X=0 & Y = 1) p(X=1 & Y = 0)]*[p(X=1 & Y = 1) p(X=0 & Y = 0)]-1} [1] X and Y represent pairs of concepts. Obtaining the association grade among X and Y requires calculating the p(X = 1 & Y = 0) value by determining the joint probability that X appears when Y does not appear in a SAM group. Furthermore, p(X = 0 & Y = 1) and p(X = 0 & Y = 0) are calculated similarly. Calculating the p(X = 1 & Y = 1) value considers a hierarchical modulation of the M value in each SAM group and their interconnectivity through the neuro-computational network. These calculations served to configure a connectivity matrix (SASO Matrix) (Semantic Analyzer of Schemata Behavior) (Lopez & Theios, 1992) useful for the semantic analysis of schematic behavior. Semantic Priming Study: The Mental Chronometry of Academic Learning Finally, a semantic priming study determined the degree of learned information consolidation in the student's memory. The study presented pairs of words (prime-target) that could have different relationship kinds (associative vs. schematic vs. unrelated). Figure 2 illustrates the experimental sequence. Figure 2. The Sequence of an Experimental Test of the Semantic Priming Study The experimental sequence consisted of a black dot in the computer screen center presented for 500 ms, after the first concept appeared (prime) for 250 ms. Finally, the last word (target) appeared and remained on the screen until the participant performed the experimental task, which consisted of reading each pair of words (prime-target) silently and International Journal of Educational Methodology  217 then making a lexical judgment about the target. The participant categorized the last word as ‘word’ or ‘no word’. The study duration ranged from 7 to 9 minutes, depending on the participant's performance. Data Analysis First, the authors performed a multidimensional scaling analysis on the first study's data to inspect the organization of the participants' schema. After, they carried out a computational simulation through a constraint-satisfying neural network. This kind of tool facilitates the observation of knowledge schema activation and co-activation behavior. Finally, a third analysis explored the chronometric patterns of information processing related to the evaluated course. The authors obtained the IRT (the time each student used to recover each definer of the NSN from their memory). Following this, they applied a mixed ANOVA on the RT obtained in both the experimental and control groups’ semantic priming study to determine the consolidation level of information in the student’s memory at the beginning and the end of the course. The authors carried out analysis of variance with consideration that the level of measurement of the dependent variable was ratio. The observations had a condition of independence as the experimental conditions and the participants were assigned randomly for this study. Regarding equinormality in the data, the QQ plot showed that the data distribution was normal, while Levene's test showed that the variances were equal across the experimental conditions for both the control group and the experimental group. Results The authors organized the results in three dimensions: the first describes the content and organization of the knowledge schema on computational cognition. The second is the observation of the schematic dynamics and connectivity pattern of the NSN. The last dimension examines the conceptual accessibility and consolidation of the information in the learned schema. Schema's Content and Organization Analysis A multidimensional scaling analysis revealed that the experimental group presented changes in the conceptual organization and quality of the definers related to computational cognition. While they used concepts from a mixed psychology and technology general schema in the course beginning, toward the course's end, the students organized their concepts in two conceptual axes under a specialized schema related to computational cognition (Figure 3). One axis involved HIP (Human Information Processing) and PDP (Parallel Distributed Processing) concepts. The second axis incorporated concepts related to the body-mind duality issue from a cognitive perspective, otherwise known as the hardware-software metaphor. Figure 3. Definers’ Conceptual Organization at the Course Beginning and the End for the Experimental and Control Group 218  MORALES-MARTINEZ ET AL. / Cognitive Technology to Evaluate the Academic Learning In contrast, although the control group incorporated new concepts towards the course end, they belonged to a general psychology schema, and their organization remained relatively similar at the course beginning and end. Connectivity Patterns and Schema Dynamics Academic learning expression also includes the formation of new connections, the loss of existing ones, or the change in connection weights. Figure 4 shows the changes in the initial and final connections between the targets evaluated by the experimental and control groups. Notice in the figure that the experimental group increased the number of connections on different targets. For example, at the beginning of the course, they connected computation only with three targets (computational mind, Turing machine, and HIP); after the course, participants formed six new connections (von Neumann, long-term memory, working memory, connectionism, memory, mind) with this same concept for nine final connections. Furthermore, they lost conceptual nodes, disconnected long-term memory, and von Neumann, and formed a connection between the Turing machine and computation. Also, participants increased or decreased the connectivity strength among different targets. For example, the computational mind-von Neumann pair strengthened their relationship by increasing the number of common definers from one to five toward the course end. The control group gained and lost connections and changed connection strength among different target pairs. However, the control group had very little change in concepts such as HIP compared to the experimental group's performance. Figure 4. Connectivity Graph Obtained Before and After the Course by the Experimental and Control Group Note. A color node represents each target, and the node's size is related to the connectivity degree. The definers' number connecting each target pair is over the linking line. The more definers connect each target pair, the darker the line. On the other hand, the authors analyzed two aspects to explore the schematic behavior: the levels of schema activation before and after the course and the co-activation pattern among definers. In both cases, the SASO matrix helped graph the schema behavior. The results indicated that only the experimental group's schema activity was significantly modified. By contrast, in the control group, the activation remained similar between the initial and final measurements (Figure 5). The experimental group increased its conceptual activation level at the course end, as shown in Figure 5, where greater activation is presented with a red and higher graph position. International Journal of Educational Methodology  219 Figure 5. Experimental and Control Group Surface Plots The authors illustrated the co-activation schema patterns by activating common concepts with the highest M values in the experimental group (Figure 6) and these concepts experienced a meaning change at the course end. In this regard, the brain went from a vision directly associated with memory to a cognitive science view (neural networks). The computer passed from a classical computation schema toward a human information processing vision. However, these concepts did not experience any change in the control group. 220  MORALES-MARTINEZ ET AL. / Cognitive Technology to Evaluate the Academic Learning Figure 6. Patterns of Co-Activation for Two Definers with the Two Highest M Values in Both Groups Mental Chronometry and Consolidation of the Schema A qualitative analysis of the experimental group's IRT revealed that the relation among conceptual accessibility of definers in their M values is negative (Figure 7). The correlation increased significantly for the experimental group at the end of the course (r = -.31) compared with the initial NSN (r= -.16). This means that after students learned computational cognition schema, they accessed definers with the highest M value in a shorter time than when the concepts obtained less semantic relevance. Similarly, the control group exhibited a correlation change from the initial NSN (r = -.42) to the final NSN (r = -.55). Figure 7. Relationship Between the Inter-Response Times and the M Values Obtained for All the Definers by the Experimental and Control Groups at the Course Beginning and End International Journal of Educational Methodology  221 The experimental group changed in the pattern of access time and semantic relevance of definers (Figure 7). However, a student’s t test for dependent samples, performed on the access times on the common definers obtained before (M = 30, SD = 9.6) and after the course (M = 27, SD = 3.1), indicated that there was not a statistically significant difference. The scarcity of common definers and the high variability in the IRT from the course beginning likely influenced this result. On the other hand, a student's t-test was also applied to the M values obtained in the common definers before (M = 62, SD = 23) and after (M = 148, SD = 50) the course, and the difference was statistically significant (t(8) = -4.17, p = .004). On the other hand, the control group showed greater changes in the M values than in the common definers' IRT distribution. In this regard, a student’s t-test (t(42) = .85) indicated that there are no significant differences between the IRT means obtained for the commons definers at the course beginning (M = 29, SD = 10) and the end (M = 28, SD = 9.2). However, there was a significant difference (t(42) = -2.77, p = .008) between the M value means of the initial (M = 65 SD = 37) and final NSN (M = 56 SD = 24). Figure 8 shows these changes in the interaction pattern between IRT and M-value, only for the common definers between the initial and final NSN. Figure 8. Relationship Between the Inter-Response Times and the M Values Obtained for the Common Definers by the Experimental and Control Groups at the Beginning and End of the Course The final chronometric analysis comprised a mixed ANOVA of 2(Group: Experimental vs. Control) x2(Course time: Start vs. End) x3(Semantic relationship: Associative vs. Schematic vs. None) on reaction times obtained in the semantic priming experiment (Table 1). The significance level was at p≤.05. The analysis included only the reaction times for the correct answers for 45 participants in each group. The authors eliminated those participants who did not obtain at least 70% correct answers in the experimental conditions in any of the two measurement moments.