Computational Concepts and their Assessment in Preschool Students: An Empirical Study

Computational Concepts and their Assessment in Preschool Students: An Empirical Study Marcos Jiménez1,2 ·María Zapata-Cáceres1·Marcos Román-González3·Gregorio Robles1· Jesús Moreno-León4·Estefanía Martín-Barroso1 Accepted: 26 July 2024 / Published online: 9 October 2024 © The Author(s) 2024 Abstract Computational thinking (CT) is a multidimensional term that encompasses a wide variety of problem-solving skills related to the field of computer science. Unfortunately, standardized, valid, and reliable methods to assess CT skills in preschool children are lacking, compromising the reliability of the results reported in CT interventions. To surpass this limitation, we validated in a sample of 700 preschool students (5–6 years old) the Beginners Computational Thinking test Short-Form (BCTt-SF), an unplugged 12-item instrument that measures three of the most common computational concepts assessed in preschool research: sequences, loops, and conditionals. The theoretical model underpinning the BCTt-SF was supported by dimensionality assessment, which suggested that preschool students can be distinguished in terms of four specific abilities (i.e., sequences, simple loops, nested loops, and conditionals) and that all of these abilities were related by a general factor. We modeled this hierarchical structure with a bi-factor model that presented excellent psychometric properties, from good statistical fit indices to adequate reliability of the general ability. To take full advantage of this model, we created an online application in the Shiny platform (https://computationalthinkingtests.shinyapps.io/SF-BCTt/) for the seamless scoring of examinees by any teacher or researcher who uses the BCTt-SF to assess CT skills in preschool children. Finally, we demonstrated how the BCTt-SF can be used to test the impact of educational interventions for improving CT skills in preschoolers. Keywords Computational thinking ·Early years education ·Evaluation methodologies ·Psychometric validation · 21st century abilities BMaría Zapata-Cáceres [email protected] Marcos Jiménez [email protected] Marcos Román-González [email protected] Gregorio Robles [email protected] Jesús Moreno-León [email protected] Estefanía Martín-Barroso [email protected] 1Universidad Rey Juan Carlos, C/Tulipán s/n Móstoles, 28933 Madrid, Spain 2Universidad Autónoma de Madrid, Madrid, Spain 3Universidad Nacional de Educación a Distancia, Madrid, Spain 4Universidad de Sevilla, Seville, Spain Introduction Many instances of problem solving in daily life require mastering abilities closely related to terms used in computer science. For example, pattern recognition is an important skill for driving a vehicle to identify obstacles in traffic flow and adjust the route accordingly, whereas interpreting conditional rules (e.g., stopping if encountering a stop sign or at a yield sign while other vehicles keep approaching from either direction) are also essential aspects for driving that are reminiscent concepts from computer science. However, the application of computational thinking (CT) skills is not restricted to highly standardized activities. Contrary, it comprises “the thought processes involved in formulating problems so their solutions can be represented as computational steps and algorithms” (Aho, 2012). As such, these processes can be transferred to nonprogramming domains and more nuanced situations (Weintrop et al., 2021). For instance, a person who is planning a vacation must break Journal of Science Education and Technology (2024) 33:998–1020 999 the problem down into smaller, manageable parts (e.g., destination, transportation, accommodations, activities, budget, timing), which is an ability termed decomposition (Shute et al., 2017). The fact that everyone uses CT abilities on a routinely basis and with different degrees of success has motivated many educators to suggest that CT should be a compulsory subject as worthy as reading, writing, and arithmetic (Wing, 2006) and that it should be assessed at all levels of education (El-Hamamsy et al., 2022b; Relkin & Bers, 2021;Stamatios, 2021), including preschool grades (Georgiou & Angeli, 2019; Stamatios, 2022). As a consequence, there has been a significant progress over the last years to promote CT in compulsory education (Kong, 2016; Manches & Plowman, 2015; Saxena et al., 2020; Wang et al., 2020; Weintrop et al., 2021), starting as early as primary school in some countries, such as New Zealand, South Korea, Canada, or Singapore, and in preschool, such as in Australia and Japan (Bocconi et al., 2016). With the advent of this new academic curricula in which students acquire CT skills, instruments for monitoring their learning progress have been proposed recently. The assessment of CT skills is a highly valuable endeavor because it helps to assess strengths and gaps in the curricula, to identify children who need further support, and to provide feedback to teachers on the effectiveness of their teaching methods. Additionally, assessing CT skills can also serve to identify areas where students are excelling, prompting opportunities for enrichment and challenge for these students (Román- González et al., 2018). Alternatively, it may promote social equity by helping children with disabilities or special needs to prepare them for higher levels of education (Taylor, 2018). Up to date, instruments for measuring CT skills span a wide audience, ranging from preschoolers to secondary school students (El-Hamamsy et al., 2022b; Relkin & Bers, 2021; Relkin et al., 2023; Román-González et al., 2018; Zapata-Cáceres et al., 2020,2021). Notwithstanding, the development of rigorous testing procedures for the former group remains scarcer. This is a problem because an instrument designed for a specific age range is likely to be too difficult for those immediately younger, differing also in its capacity for disentangling students with different levels of skill. And more worrisome, a recent review of assessment methods in CT research for preschool children found that most of these methods were not standardized but created adhoc and did not undergo a proper validation and reliability analysis (Tang et al., 2020). To overcome these limitations, we made a psychometric adaptation for preschool students of the Beginners Computational Thinking test (BCTt; Zapata-Cáceres et al., 2020), an instrument originally designed for children aged 5 to 9 years old that has been validated for measuring the understanding of three computational concepts in lower-primary school: sequences, loops, and conditionals. Sequences are sets of instructions or steps that need to be taken in turn; loops are rules to run the same sequence a given number of times, either by repeating each time a sequential term (i.e., simple loop) or by repeating a number of times a block of sequential terms (i.e., nested loop). Finally, conditionals are indications to perform an action when either a specific circumstance is satisfied (i.e., if-then), when a condition is satisfied and another action when it is not (i.e., if-then-else), or as long as a condition remains satisfied (i.e., while). In the BCTt, these three broad computational concepts are put into test in 25 items, categorized as either grid format (21 items) or canvas format (4 items). In the grid format items, the examinees are instructed to move an object (i.e., a baby chicken) across a grid to reach a target (i.e., the mother of the baby chicken) by spotting the correct sequential pattern from a list of four alternatives. Figure1provides an illustrative example of a typical grid item format in the BCTt, encompassing each of the aforementioned dimensions. Conversely, in canvas format items, the examinees are presented with a simple path drawing over a canvas and must select the correct sequential pattern that corresponds to the given path from another list of four alternatives. An important note is that, although the BCTt is mainly focused on the assessment of these computational concepts (e.g., sequences, loops, and conditionals), it is not limited to them. Contrary, computational practices such as testing and debugging are also required to solve the questions of the BCTt because the examinees need to inspect the four response alternatives (i.e., A, B, C, and D) to spot the erroneous ones. The BCTt bears the potential to be applied in preschool settings because it was found to be most suited for lowpercentile students in primary school (Zapata-Cáceres et al., 2021). However, the average time of completion in lowerprimary school students was about 40min (Zapata-Cáceres et al., 2021). Therefore, reducing the test length is necessary for preschool children to complete it, as they need a longer time to answer most of the items and are more prone to fatigue. Such, the aim of this study is three-fold: (1) First, to identify the nonoverlapping computational concepts that the students can differentially understand and upon which can be differentiated. (2) Second, to select the subset of items from the BCTt that consistently measures the target computational concepts in preschool students, discarding those that are tangential or measure spurious features. We also intend to shorten further the test by selecting the items that best encompass all the computational dimensions while retaining adequate psychometric properties of reliability and content validity, creating the Short-Form of the Beginners Computational Thinking test (BCTt-SF). 1000 Journal of Science Education and Technology (2024) 33:998–1020 Fig. 1 Depictions of four items belonging to the BCTt, each one taken from a different computational concept block. Upper-left: Sequence block; bottom-left: Simple Loop block; upper-right: Nested Loop block; bottom-right: Conditional block (3) Third, to create an online application where teachers can upload the responses of their students and automatically obtain the corresponding scores on all the computational dimensions assessed by the BCTt-SF. Computational Thinking in Preschoolers Although no consensus definition of CT currently exists, there is agreement in that CT is a multidimensional term that encompasses a wide variety of problem-solving skills, including logical thinking, development of algorithms, data representation and manipulation, computer programming, and more (Shute et al., 2017). In other words, CT is not just about solving problems in a computational environment, but applying thought processes that lead to problem evaluation, simplification, and elaboration of strategies to find a solution using a set of skills that are related to computational science. However, the existence of multiple definitions, categorizations, and assessment proposals of CT abilities makes difficult for educators to decide what specific competences should be addressed in a given age group and how to do it. For instance, while some authors warn about the scarcity of research with regards to the specific skills that preschoolers candominate (Relkin & Bers, 2021), others remark the lack of consensus about what computational concepts and practices should be taught to children at these early ages (McCormick & Hall, 2022). According to Piaget’s theory of cognitive development, preschool children fall under the preoperational stage of development that spans between 2 and 7 years old. This is a period when children acquire language and symbolic thinking. They learn to represent objects and events mentally without their physical presence and through images and words, prompting the appearance of abilities such as sequencing, pattern recognition, and algorithm design (Saxena et al., 2020). In this vein, there exists some research in preschool children in relation to the specific concepts of sequences, loops, and conditionals that the BCTt measures. Specifically, there is consensus in that preschoolers can solve with accuracy simple loops problems (Angeli & Valanides, 2020; Bers et al., 2014; Saxena et al., 2020), although they struggle more than with complex sequences (Elkin et al., 2016). In any case, it is considered that this ability is already mastered in early 123 Journal of Science Education and Technology (2024) 33:998–1020 1001 primary school (Relkin & Bers, 2021). With regards to their performance in solving problems with loops, research also shows that they can understand and apply with success this concept (Angeli & Valanides, 2020; Bers et al., 2014; Relkin &Bers,2021). Lastly, there is also evidence that preschoolers can solve certain tasks involving conditional statements (Andrews et al., 2012; Diamond et al., 2005). In fact, it seems that they can do it much before than what it was previously thought (Ling et al., 2021), indicating that it is a concept that preschoolers can understand at an age of 3 years old. Test Development in Computational Thinking Research for Preschoolers A systematic review conducted by Bakala et al. (2021) confirmed that sequences, loops, and conditionals were, in fact, the computational concepts most commonly assessed in the literature for the purpose of testing educational interventions. However, despite the recent development of many appealing educational programs and curricula for promoting the learning of CT skills in preschool children, the lack of valid instruments for testing them remains as one of the biggest problems in the field (Bakala et al., 2021; Tang et al., 2020). Moreover, in order to make an objective comparison between the many different educational interventions that have been developed in the field, it is necessary to operationalize the computational concepts and practices that are relevant for different age groups and create standardized instruments accordingly (McCormick & Hall, 2022). Thus, we currently do not know to what extent the rubrics, tasks, and tests that conform these assessment methods measure the intended computational constructs and, ultimately, if they are valid for the purpose of scoring students. As a consequence of this lack of validated instruments, the fair comparison of different CT interventions is neither possible. Moreover, the samples recruited in these studies were generally small, precluding even the application of reliable psychometric analyses. Another problem is that most of the reviewed instruments were plugged (i.e., they involved tasks where students needed to interact with electronic devices, write or interpret code, program robots, and perform other activities that required specific computational knowledge). Unfortunately, these kinds of plugged assessments have drawbacks. When using technological environments to measure CT skills, children who lack the necessary coding knowledge to participate cannot be assessed, prompting the risk of conflating CT abilities with programming skills. Moreover, some children may outperform others because of more familiarity or prior exposure to technological environments and a more developed fine-motor functioning. An additional disadvantage of plugged methods for assessment is that they are costly because they require materials (e.g., physical kits) and the individual observation of each child. Furthermore, plugged tasks are often assessed with portfolios and rubrics, which can also be problematic due to the subjectivity of the teacher who scores the behavior of the children. For these reasons, many of the new psychometric instruments that are being developed take an unplugged design based on traditional tests (El-Hamamsy et al., 2022b;Li et al., 2021; Relkin & Bers, 2021; Relkin et al., 2020, 2023; Román-González et al., 2015; Zapata-Cáceres et al., 2021; Zhang & Wong, 2023). Consider the TechCheck (Relkin et al., 2020), an unplugged 15-item test designed for preschool students that measures six different computational concepts and practices: algorithms, modularity, debugging, hardware/software, control structures, and representation. The TechCheck does not require any coding knowledge and can be easily scored. In addition, it was adapted to the population of preschool students using a sample of 395 students, yielding the revised version TechCheck-K (Relkin & Bers, 2021; Relkin et al., 2023). Another example is the BCTt (Zapata-Cáceres et al., 2021). As pointed out by Bakala et al. (2021), instruments like the BCTt are advantageous because many of the activities that were assessed in the literature were closely related to the tasks used during the educational program, whereas the BCTt offers measures that are independent from the devices used during the intervention, providing information about the transferable properties of CT skills. Moreover, the BCTt is an instrument created from adaptations of previous tests widely used in higher education levels that have undergone validity and reliability assessments (Table 1). In fact, we are not the first authors who use the BCTt to create an adaptation for students of a different age group. The BCTt has already been successfully adapted for upper-primary school students by increasing item difficulty, giving raise to the competent Computational Thinking test (cCTt; El-Hamamsy et al., 2022a). As such, the theory underpinning the BCTt is well consolidated, and the BCTt itself serves as a benchmark for other tests as well as a template for the development of new tests (El-Hamamsy et al., 2022a). To determine the internal validity of the BCTt and to accomplish a reliable adaptation of the BCTt with a suitable test length for preschool students, we pursued the three objectives outlined above: First, we determined which computational concepts can be distinctly assessed in a sample of preschool students. Second, we undertook a purification process in which screening out the items that do not measure the intended computational concepts. As a result, we selected the most informative items while maintaining adequate validity and reliability properties, resulting in a shortened version of the test (BCTt-SF). Third, we provided an online applica- 1002 Journal of Science Education and Technology (2024) 33:998–1020 Table 1 Review of Computational Thinking tests directly related to the BCTt-SF in chronological order Test Format Target group CT concepts Psychometric assessment Sample CTt (Román- González et al., 2015,2017, 2018; Guggemos et al., 2022) 28 items Secondary school (10-16 years old) Sequences, Finite Loops, Loops over a condition, Conditionals (i.e., If-then, Ifelse, and While statements), and Functions Expert validation, Reliability estimation within CTT (i.e., Cronbach alpha) and Item Response Theory, Criterion validity, Predictive validity 400 students (Román- González et al., 2015) 1251 students (Román-González et al., 2017) 314 students (Román-González et al., 2018; Guggemos et al., 2022) BCTt (Zapata- Cáceres et al., 2020) 25 items lower-Primary school (5-7 years old) Sequences, Simple Loops, Nested Loops, and Conditionals (i.e., If-then, If-else, and While statements) Reliability estimation within CTT (i.e., Cronbach alpha) 299 students cCTt (El- Hamamsy et al., 2022a) Three versions of 25, 17, and 15 items upper-Primary school (7-9 years old) Sequences, Simple Loops, Nested Loops, Conditionals (i.e., If-then, If-else, and While statements), Complex (i.e., combinations of the previous concepts) Expert Validation, Model fit assessment with CFA, Item calibration and reliability estimation within IRT (i.e., item information), Reliability estimation within CTT (i.e., Cronbach alpha) 1519 students BCTt-SF (current study) Selection of 20 items Final version of 12 items Preschool students (4-6 years old) Sequences, Simple Loops, Nested Loops, Conditionals (i.e., If-then and If-then- else) Dimensionality assessment, Model fit assessment with CFA, Item calibration and reliability estimation with MIRT (i.e., item information), and automatic test scoring via an online application 700 students Note. CTt: Computational Thinking test; BCTt: Beginners Computational Thinking test; cCTt: competent Computational Thinking test; BCTt-SF: Short-Form of the Beginners Computational Thinking test; CTT: Classical Test Theory Journal of Science Education and Technology (2024) 33:998–1020 1003 tion for the automatic and seamless scoring of any preschool examinee. Methods In this section, we describe the details of the data collection and the psychometric analyses that we carried out to develop the adaptation of the BCTt for preschool students. Participants and Data Collection A sample of 700 preschool students aged 5 to 6 years old recruited from 8 classrooms across several Spanish regions participated in the current study. The sample was divided in two groups: quasi-experimental and control. The quasi-experimental group encompassed 595 children whose teachers subscribed voluntarily to an educational program promoted by the Spanish Ministry of Education and Vocational Training.14 The aim of this project was investigating the impact of the unplugged CT activities in the development of the CT skills of children. The educational intervention, termed School of Computational Thinking and Artificial Intelligence, included a training phase for the teachers (i.e., from 13 November 2020 to 1 February 2021) and a curriculum implementation phase for the children (i.e., from 1 March to 31 May 2021). The training phase was composed by four blocks. In the first block, CT was introduced to the teachers, who learned the definitions of several concepts such as sequences, algorithms, logical thinking, abstraction, decomposition, parallelism; in the second block, they completed different unplugged activities involving these concepts to acquire familiarity with them. In the third block, they learned how to analyze the didactic value of the unplugged activities using Cody & Roby,2a game in which Cody (i.e., the programmer) creates instructions that Roby (i.e., the robot) must execute. The final block was dedicated to the presentation of unplugged activities for the teaching of Artificial Intelligence (Touretzky et al., 2019). In total, the teachers dedicated 30 h to complete all of these training blocks. Once the training phase was completed, the teachers implemented their custom educative intervention in at least five lectures during the following months. This intervention was integrated into standard subjects of the curriculum (i.e., mathematics, linguistics, natural and social sciences, physical education, foreign language, and music) following the guide proposed by AI4K123(Nishida et al., 2009). In this manner, the CT activities also contributed to the learning of 1https://intef.es/ 2http://www.codeweek.it/cody-roby-en/ 3https://ai4k12.org/ specific content from these subjects. During the implementation phase, the children participated in five to ten CT teaching sessions that included unplugged CT activities. Only the children whose teachers completed both phases of the program were considered in the study. On the other hand, the control group was composed by 105 students recruited with the help of the Education Departments of different Spanish regions. The teachers in the control group also participated voluntarily in the study, but they did not undergo the educational intervention. The only requirement for the students in the control group to participate in the study was the absence of unplugged CT activities in their daily teaching during the current and previous academic courses. After completing the implementation phase in the participating schools, all students, both in the quasi-experimental group and the control group, took the BCTt to measure their CT skills. The BCTt is an instrument that has been widely used in primary school students (Zapata-Cáceres et al., 2020; Zapata-Cáceres & Fanchamps, 2021) and has been found to display good reliability in terms of internal consistency, with a Cronbach’s alpha coefficient of .824 (Zapata-Cáceres et al., 2020). As such, the BCTt may be a reliable measure to assess the CT skills of the students in the current study. Regarding the test administration, the BCTt was taken during a class period, with students having a total of 45 min to answer the questions, in addition to a few minutes of prior explanation about the test scoring. The BCTt is a paper-based test, so each student had an individual printed copy of the test. To ensure that the instructions given to the students before taking the test were as similar as possible in all the schools, a video was recorded4explaining each type of question of the test. This video was made available to teachers so they could show it to their students in class just before they took the test. The teachers were also provided with an action protocol and a document with answers to frequently asked questions. In the end, the educational intervention used in our study was not designed to teach the children any particular concept included in the BCTt. Consequently, we expect that the results from the BCTt indicate their overall CT learning, rather than being affected by specific knowledge acquired from test preparation. Data Analysis and Psychometric Validation We first inspected the data with descriptive statistics (Table 2). These included the proportion in which every option to each item was selected and the cumulative missingness at each item. Table 2also displays the qualitative characteristics of the items composing the BCTt, which includes their format (i.e., grid vs. canvas) and the particular mistakes that produce the selection of erroneous response alternatives (i.e., 4https://vimeo.com/544462776/34e6be254a 1004 Journal of Science Education and Technology (2024) 33:998–1020 Table 2 Item features and error profiles for each response alternative. The proportion in which each alternative was chosen is also presented within parentheses Response alternatives Missingness Error Profiles Block Q Format A B C D Prop WD Diff 1 2 3 4 5 6 Sequences 1 Grid (3 ×3) (.89) 1 (.05) 6 (.03) 1 (.03) .00 .00 −Wrong Insufficient Fails to avoid Fails to pick −Other error 2 Canvas 1 (.05) 1 (.05) 1 (.05) (.81) .04 .00 −direction steps the obstacle the object 3Grid(3×3) 3 (.05) 2|3 (.04) (.82) 1 (.05) .03 .00 − 4Grid(3×3) 2 (.06) (.77) 3|4 (.05) 4 (.09) .04 .00 -0.58 5Grid(4×4) 3|4 (.07) (.77) 3 (.07) 2|3 (.07) .02 .00 -0.10 6 Canvas 1 (.06) 1 (.06) 2 (.09) (.72) .07 .01 1.06 Simple Loop 7 Grid (3 ×3) 2 (.12) (.76) 5 (.06) 5 (.02) .04 .01 3.89 Wrong Insufficient Fails to avoid Fails to pick Leaves Other error 8Grid(3×3) (.74) 5 (.10) 5 (.08) 1 (.04) .05 .01 0.89 direction steps the obstacle the object the grid 9Grid(3×3) 3 (.11) 3 (.06) 6 (.05) (.73) .05 .02 2.19 10 Grid (4 ×4) 2 (.09) 2 (.05) (.75) 2 (.04) .07 .03 0.12 11 Grid (4 ×4) 4 (.18) (.63) 6 (.10) 6 (.04) .06 .03 - Nested Loop 12 Grid (4 ×4) (.61) 2 (.13) 2 (.05) 2 (.13) .08 .03 1.56 Wrong Insufficient Fails to avoid Fails to pick Leaves Other error 13 Canvas 1 (.09) 1 (.06) (.68) 1 (.09) .09 .04 0.09 direction steps the obstacle the object the grid 14 Grid (4 ×4) 3 (.26) (.41) 3|5 (.05) 2 (.17) .11 .05 0.72 15 Grid (4 ×4) 4 (.32) 2 (.12) (.23) 2 (.21) .13 .07 7.81 16 Grid (4 ×4) 4 (.22) 4|5 (.15) 2 (.09) (.38) .17 .10 -0.24 17 Canvas (.69) 1 (.06) 1 (.04) 1 (.04) .16 .11 1.59 18 Grid (4 ×4) 2 (.23) 3 (.15) (.33) 4 (.11) .17 .11 3.67 Journal of Science Education and Technology (2024) 33:998–1020 1005 Table 2 continued Response alternatives Missingness Error Profiles Block Q Format A B C D Prop WD Diff 1 2 3 4 5 6 If-then 19 Grid (3 ×3) (.51) 1 (.11) 2 (.08) 5 (.11) .19 .13 2.35 Follows a Wrong direction −Other error 20 Grid (4 ×4) 1 (.27) 2 (.14) (.27) 2 (.25) .25 .15 1.75 simple sequence If-then-else 21 Grid (4 ×4) 2 (.15) 3 (.21) 2 (.14) (.25) .25 .18 0.30 −Wrong direction Insufficient steps −− 22 Grid (4 ×4) 2 (.27) (.25) 2 (.09) 2 (.11) .28 .20 -0.18 While 23 Grid (3 ×3) (.51) 2 (.11) 2 (.08) 2 (.06) .24 .22 1.80 −Wrong direction −Fails to move from − 24 Grid (4 ×4) 4 (.30) (.26) 2 (.09) 2 (.07) .28 .25 1.89 starting point 25 Grid (4 ×4) 2 (.16) 2 (.13) (.31) 2 (.10) .31 .31 - Note. Q: Question; Prop: proportion of missing responses; WD: cumulative proportion of withdrawal from the item onward; Diff: Difference between the sum score of the examinees that answered all the questions to the sum score of those who withdrew after answering item; Wrong direction: Moves in an opposite direction or to a farther place with respect to the target; Insufficient steps: The number of steps are not enough to reach the target; Fails to avoid the obstacle: Passes through the black cat; Fails to pick the object: Does not pick the flower 1006 Journal of Science Education and Technology (2024) 33:998–1020 error profiles). We identified six error profiles across all the sequences and loop blocks: Moving in an opposite direction or to a farther place with respect to the target (profile 1); taking an insufficient number of steps to reach the target (profile 2); failing to avoid an obstacle (profile 3); failing to pick a required object (profile 4); leaving the grid (profile 5); and committing another kind of error (profile 6). The error profiles identified in the items from the conditional block are more idiosyncratic and do not overlap with the previous ones. Although the error profiles will not be involved in any quantitative analysis, we included these profiles because we think they would help teachers to identify problematic patterns of reasoning in children, facilitating the design of teaching methods focused on mitigating these errors. The second step consisted of identifying the dimensionality and structural configuration of the test in the sample. Dimensionality assessment is a particular complex task since model misspecification is an important obstacle towards understanding the underlying organization of the data (Mac- Callum, 2003). In other words, all statistical models ignore many of the nuances that explain some item variance (e.g., fatigue, distraction, pressure, acquiescence). As a result, empirical data is messy, difficulting its alignment with theory even when the main features of such a theory are true. Fortunately, Exploratory Graph Analysis (EGA; Golino & Empskamp, 2017b) is a recent dimensionality tool that can be used to uncover the underlying organization of the traits underpinning the responses to a set of items. It employs a regularization procedure designed to clean these nuances from the data. More specifically, EGA fits a Gaussian Graphical Model (GGM) with GLASSO regularization (Friedman et al., 2008; Golino & Epskamp, 2017b) to estimate sparse correlations between the items (i.e., their pair-wise linear association after taking out the shared variance with the remaining items) and then applies a clustering algorithm (e.g., Louvain) to classify each item in different, nonoverlapping groups. The result of this analysis is usually visualized as a graph, where each node is an item, the node color represents a cluster, and the thickness of the edges connecting them indicates the strength of their linear relationship after conditioning on all the remaining items. Once this first dimensionality analysis was completed, a hierarchical version of EGA was performed (hierEGA; Jiménez et al., 2023). A score on each cluster was computed for each participant, and the GGM with Louvain clustering was applied again using these scores as input. As a result, the clusters computed in the first step (i.e., first-order clusters) were themselves clustered in higher-order clusters. This last step was used to identify the hierarchical organization of the data. In other words, we checked whether the first-order clusters were embedded within one or more higher-order clusters. Overall, we expected to find well-defined first-order clusters composed by items related to sequences, loops, and conditionals, which are the three broad blocks of computational concepts used to design the items of the BCTt, as well as a single higher-order cluster (i.e., general factor) accounting for the relationships between them. To obtain a more fine-grained analysis of the items and to develop a proper item scoring method, Multidimensional Item Response Theory (MIRT) was used. More concretely, we used a bayesian Three-Parameter Logistic (3PL) bi-factor model to estimate the probability of succeeding an item. According to the bayesian 3PL bi-factor model, the conditional probability of an examinee correctly scoring an item jis defined by the inverse logit function (Cai et al., 2011), pj=cj+1−cj 1+exp(−[θgαgj +θsαj+dj]+G×ηj),(1) where cjis a parameter indicating the basal probability of succeeding item jby guessing, θgand θsare the values of the general and specific abilities with respective slope or discrimination parameters αgj and αj, and djis a location parameter for item j, usually termed the easiness parameter. Gis an indicator variable that is 1 if the examinee belongs to the quasi-experimental group and 0 otherwise. This manner, the ηjindicated the difference in location between the item characteristic curves of the quasi-experimental and control groups, taking into account the effect of the educational intervention in the quasi-experimental group. Thus, these ηj parameters are akin to the ones that are commonly used to test for uniform differential item functioning. To ascertain that the model represented accurately the observed data, we performed posterior predictive checks of the total score distribution for the quasi-experimental and control groups and for the probability of succeeding each item. Regarding the model parameter estimates, we flagged guessing estimates higher than .25, as they would imply that the basal probability of correctly answering the item is compatible with choosing a response alternative by chance. Taking together the results of the analyses, we selected three items from each block of computational concepts, focusing on the items that were most informative for the general factor across a range of abilities that were not too high or too low. We also considered desirable that the items loaded on the specific dimensions and did not have high guessing probabilities. The selected items conformed the BCTt-SF, whose underlying bi-factor structure is anticipated in the diagram of Fig. 2. To assess the adequacy of the final 12-item selection, we repeated all the psychometric analyses on the items comprising the BCTt-SF. Finally, to take full advantage of the final statistical model, we created an application in the Shiny platform to automatically score examinees from their patterns of success/failure responses (https://computationalthinkingtests.shinyapps.io/ 123 Journal of Science Education and Technology (2024) 33:998–1020 1013 Second, the pattern of missing responses was mostly driven by test withdrawal probably due to both fatigue and discouragement of low-performing examinees. This could introduce some misspecification in the statistical model that we fitted. A possible solution to this could be to generalize the bi-factor model to explicitly model the probability of withdrawal and relate it to the ability of the examinees. Another problem of progressive test withdrawal is that it could prompt systematic correlations between items that are proximal in the test, a phenomenon that was also ignored. Unfortunately, the current study neither provides information on how the results can be linked to specific teaching methods and the teachers’ characteristics. Third, we did not elucidate whether the general factor of computational concepts assessed by the BCTt-SF actually corresponds to a measure of general intelligence (i.e., discriminant validity). Previous research showed that CT skills are correlated with measures of intelligence like spatial, reasoning, and problem-solving abilities (Román-González et al., 2017,2018) and also relate to students’ numerical and verbal abilities (Tsarava et al., 2022). A final limitation of this work is that the BCTt-SF is focused on computational concepts and practices and the assessment of computational perspectives is lacking (Brennan & Resnick, 2012). Computational perspectives are abilities that allow children to share information and express ideas, allowing them to teach others, interact, collaborate, discuss, and bring into question the surrounding technological world. In other words, these abilities are the most socially oriented as they bear the purpose of entertaining, educating, engaging with others, and critically question what and how products can be improved. Eventually, the achievement of these goals will yield computational fluency, which is the capacity to understand computational concepts and applying their meaning to express oneself creatively and effortlessly with digital technologies (Stamatios, 2021). Therefore, additional methods for assessing the progress of computational perspectives and computational fluency are required to get a broader picture of CT skills (Román-González, 2019). Despite these limitations, the current study has several strengths: we used a sample of 700 children, the largest up to date in any CT research with preschool children. This allowed us to study the competence of children in the most common computational concepts assessed in the literature, named sequences, loops, and conditionals. We replicated previous results that reported that children can solve with high accuracy problems involving simple and complex sequences and we also found that they performed well with simple loops, but presented more difficulties to understand nested loops and conditionals. We also carried out several psychometric analyses to demonstrate the internal validity and adequate reliability of both the 20-item version of the BCTt and the 12-item version composing the BCTt-SF. Additionally, it is crucial to highlight that the educational intervention implemented in our study was not tailored to specifically target the content covered in the BCTt. Therefore, the results obtained from the BCTt should reflect the generalizable properties of CT learning rather than being influenced by specific knowledge gained through test training. The BCTt-SF can serve as a valuable tool to assess the effectiveness of new educational interventions and to compare them. It can provide insights into the efficacy of different approaches in fostering computational concepts. It is important to note that the BCTt-SF can be used to test the efficacy of new educational interventions or compare them in an objective way. It should be noted that the unplugged activities carried out during the educational intervention were not specifically designed to target the content covered in the BCTt (e.g., identifying patterns in a grid). In other words, the higher scores observed in the quasi-experimental group cannot be attributed to the similarity between the unplugged activities of the intervention and the BCTt items. Empirical studies concerning preschool students often use ad-hoc methods for assessing the computational thinking skills of the children (Bakala et al., 2021; McCormick & Hall, 2022; Tang et al., 2020). Given the lack of well-validated instruments for measuring these abilities in preschool students, the BCTt-SF sets an unprecedented benchmark towards the rigorous scoring of preschool students and for testing and comparing educational interventions in their understanding of sequences, loops, and conditionals (El-Hamamsy et al., 2022a; Vourletsis & Politis, 2024). Whereas multiple tests exist for measuring CT skills in lower-primary school (Li et al., 2021; Relkin et al., 2020; Zapata-Cáceres et al., 2021; Zhang & Wong, 2023), they are lacking in preeschool settings and do not cover the aforementioned computational concepts (Relkin & Bers, 2021). Researchers and practitioners will benefit from this new assessment instrument for several reasons: (1) the BCTt-SF is a short yet reliable instrument; (2) the relevance and boundaries of the measured dimensions are clearly established; (3) the scoring system was designed according to the validated underlying statistical model and can be automated via an online application so that researchers and practitioners can seamlessly score their examinees; (4) the test is unplugged and independent from the activities and games commonly used in educational interventions like the Tangiblek Robotics Program (Bers et al., 2014), guarding against biases due to familiarity, practice, and coding knowledge. In future research, it would be valuable to explore additional forms of validity for the BCTt-SF, such as convergent and discriminant validity in relation to other computational thinking (CT) concepts. A potential approach could involve utilizing both the BCTt-SF and the TechCheck-K assessments in conjunction to gain a comprehensive understanding of students’ CT competence. This would enable the examina- 1014 Journal of Science Education and Technology (2024) 33:998–1020 tion of the interrelationships between all the computational concepts being measured. Another aspect worth investigating is the predictive validity of the BCTt-SF. For instance, gathering data on students’ performance in standard curriculum subjects and correlating them with their CT scores could provide insights into how CT relates to academic achievement. Finally, for researchers and practitioners planning to use the BCTt-SF in a preschool setting, it is important to administer the test in a physical format (i.e., paper) and to precede the test with examples. For each computational concept, the evaluator should read a representative item aloud, explain how to choose an option, and demonstrate the reasoning behind the correct choice. We also recommend allowing ample time for the examinees to read and answer all the test items. Preschoolers may sometimes lose patience or motivation or may simply withdraw from the test due to fatigue. Therefore, it is crucial to encourage examinees to answer all the items, even if they are unsure of the correct answer. This approach is recommended because the statistical model that we used does not accommodate missing responses, so all answers must be coded as correct or incorrect. This is not problematic, as the model accounts for the likelihood of correct answers occurring by chance. Conclusion An important component of the CT learning curricula is the adequate monitoring of the student progress, for which standardized, valid, and reliable psychometric instruments are required. Unfortunately, instruments with such properties for the assessment of CT skills are lacking in preschool settings. For the first time, we explicitly addressed this issue by developing a 12-item test (BCTt-SF) in a sample of 700 preschool students, obtaining excellent internal validity and reliability properties. Using the best items from the BCTt, we confirmed four well-defined, nonoverlapping factors to be independently assessed in preschoolers (i.e., sequences, simple loops, nested loops, and conditionals) and a single general factor that may reflect an overall competence in the understanding of these computational concepts. From these factors, we found that an educational intervention based on unplugged methodologies may improve the understanding of sequences and simple loops, whereas improving the understanding of nested loops and conditional statements may require more specific training at this early age. Finally, to ensure the achievement of reliable scores for preschool students, the BCTt-SF test was made freely available and can be automatically scored via an online application at https:// computationalthinkingtests.shinyapps.io/SF-BCTt/. Appendix Confirmatory Factor Analyses We sought to identify the most reliable and informative items. For this purpose, we used the graph obtained with hierEGA to define a model to be fitted with Confirmatory Factor Analysis (CFA).6Factor loadings with an absolute value lower than.30 were considered small, those between.30 and.50 were considerate medium, and larger values were deemed to be high. As the observed scores were dichotomous (1: correct response; 0: wrong response), we used the tetrachoric correlation matrix (i.e., the correlation matrix between the latent continuous scores) to fit the CFA models.7 Two types of models were estimated with CFA: a correlated factor model, in which the first-order clusters (i.e., specific factors) detected by hierEGA were allowed to correlate, and a bi-factor model, a popular method to address within-item multidimensionality by decomposing systematic item variance in a general factor that explains the commonalities between all the items and specific factors that only affect a subset of the items (Reise, 2012). If a general factor existed according to hierEGA, then we would expect medium or high correlations among all the specific factor correlations from the correlated factors model. If so, the bi-factor model could be used to quantify how much information each item contributes to measure such a general factor. Finding a general factor is very desirable because it would mean that computing a single test score reflecting computational concepts knowledge is possible. Furthermore, the reliability of such a score would be much higher than any of the specific abilities because all the items would be expected to load on this general factor. In the bi-factor model, we assessed the reliability of the general factor score with omega hierarchical (ωH; Garcia- Garzon et al., 2019), a statistic indicating the ratio of the variance accounted for by the general factor to the test variance. According to Rodriguez et al. (2015), it is considered 6Note that hierEGA is an exploratory tool concerned with the estimation of the number of factors in the data, whereas CFA is confirmatory and provides the measurement model with the regression weights of the factors on the items. As such, they do not provide redundant informationandhavebeenrecommendedtobeusedintandem(Golino& Demetriou, 2017a). 7We also used exploratory factor analysis (EFA) with the unweighted least squares estimator and target rotation to check the agreement of the loading pattern with the clustering structure of hierEGA. However, many negative cross-loadings were estimated across the structure, wrongly implying that a higher ability level in some traits is detrimental to others. Therefore, we decided to interpret only the results of the CFA models. 123 Journal of Science Education and Technology (2024) 33:998–1020 1015 that values over .80 indicate that the general factor is a very reliable source to create a single score, as most of the test variance is attributed to it. We used three fit indices to assess the adequacy of the CFA models: The unbiased root mean square residual (uSRMR; Maydeu-Olivares, 2017), the root mean square error of approximation (RMSEA; Steiger, 2000), and the Comparative Fit Index (CFI; Bentler, 1990). The uSRMR measures how close the CFA model predicts the observed correlation matrix between the items, whereas the RMSEA estimates the distance between the expected model correlation predictions and the true ones in the population. Lastly, the CFI is a measure comparing the fit of the estimated model with that of a null model where no latent factors are present. Values of uSRMR and RMSEA between .05 and .08 are usually regarded as indicating an acceptable fit and values lower than .05 indicate a good fit (Maydeu-Olivares, 2017; McDonald &Ho,2002). On the other hand, the cutoff values for the CFI are .90 or greater for an acceptable fit and .95 or greater for a good fit (Hu & Bentler, 1999). An important caveat regarding these fit indices is that the cutoffs used to assess model’s goodness-of-fit are somewhat arbitrary because they were established in specific contexts (e.g., for specific statistical models, estimators, number of items) and may not be adequate in many applied situations. Thereby, we also assessed the adequacy of the CFA models by inspecting the modification indices (MI) to detect possible sources of residual dependencies. MI are χ2statis- tics that indicate the amount of model fit improvement if a parameter that was not estimated was actually estimated in turn. These statistics may identify different types of model misspecification like residual dependencies between some items, items loading on multiple factors, and correlations between originally assumed orthogonal factors. A significant fit improvement is attained when the MI >3.84, value that corresponds to the 95th percentile of the distribution of a χ2 statistic with one degree of freedom. However, higher values are usually expected because of either sampling error (i.e., the number of false positives may be high due to the many parameters being tested) or external factors not considered in the model. Therefore, we only interpreted the MI that were significant at the 99% confidence level (MI >6.63). Overall, the CFM presented excellent absolute and relative fit indices (uSRMR[CFM]: .040 (95%CI: .031-.049); RMSEA[CFM]: .019 (95%CI: .006-.028); CFI[CFM]: .994). The CFM revealed medium-to-high factor loadings on all the items (Table 5), ranging from .54 (i.e., item 24 on the conditional factor) to .87 (i.e., item 10 on the simple loop factor). The factor correlation estimates (Fcor; Table 6) indicated that the sequence factor was strongly correlated with the ability to solve simple loops problems (Fcor =.84), which was related to the nested loop factor (Fcor =.73). The latter was also correlated with the conditional factor (Fcor =.63), creating an Table 5 Factor loading estimates and 95% confidence intervals (within parentheses) for the correlated factors and bi-factor models using the 20-item selection Correlated model Bi-factor model Item Factor Factor loading General factor loading Specific factor loading 1 Sequence 0.72 (0.61 −0.82)0.56 (0.45 −0.67)0.49 (0.31 −0.67) 30.78 (0.69 −0.86)0.62 (0.53 −0.71)0.44 (0.28 −0.61) 40.73 (0.65 −0.82)0.57 (0.48 −0.67)0.46 (0.30 −0.63) 50.72 (0.64 −0.81)0.57 (0.47 −0.66)0.44 (0.27 −0.60) 7Simple loop 0.73 (0.66 −0.81)0.72 (0.64 −0.80)0.08 (−0.14 −0.30) 80.81 (0.74 −0.87)0.82 (0.74 −0.89)0.01 (−0.21 −0.24) 90.79 (0.72 −0.86)0.75 (0.67 −0.83)0.27 (0.06 −0.49) 10 0.87 (0.82 −0.93)0.83 (0.75 −0.91)0.43 (0.13 −0.72) 11 0.61 (0.53 −0.69)0.57 (0.48 −0.66)0.28 (0.04 −0.51) 14 Nested loop 0.55 (0.45 −0.65)0.42 (0.32 −0.53)0.40 (0.21 −0.59) 15 0.61 (0.50 −0.72)0.48 (0.36 −0.60)0.35 (0.16 −0.54) 16 0.73 (0.65 −0.81)0.58 (0.49 −0.67)0.45 (0.25 −0.64) 18 0.68 (0.59 −0.77)0.55 (0.46 −0.64)0.37 (0.19 −0.54) 19 Conditional 0.61 (0.53 −0.70)0.35 (0.25 −0.45)0.50 (0.40 −0.61) 20 0.66 (0.57 −0.76)0.51 (0.41 −0.62)0.28 (0.16 −0.41) 21 0.76 (0.68 −0.84)0.45 (0.34 −0.55)0.62 (0.52 −0.72) 22 0.63 (0.53 −0.72)0.39 (0.28 −0.50)0.46 (0.34 −0.57) 23 0.68 (0.60 −0.75)0.37 (0.28 −0.47)0.61 (0.51 −0.71) 24 0.54 (0.45 −0.63)0.23 (0.11 −0.34)0.61 (0.50 −0.73) 25 0.65 (0.56 −0.73)0.34 (0.24 −0.44)0.60 (0.50 −0.70) 1016 Journal of Science Education and Technology (2024) 33:998–1020 Table 6 Factor correlation estimates in the correlated factors model using the 20-item selection Factor Sequence Simple loop Nested loop Conditional Sequence −− − − Simple loop .84 −−− Nested loop .48 .73 −− Conditional .39 .52 .63 − interdependence among the factors that could be explained by the presence of a single general factor. In fact, the smallest correlation, between the sequence and conditional factors, was medium-sized (Fcor =.39). This result, coupled with the results obtained from the hierEGA, prompted us to estimate item loadings on a general factor by means of the BFM. The BFM also presented excellent fit values (uSRMR[CFM]: .047 (95% CI: .038– .057); RMSEA[CFM]: .026 (95% CI: .017–.035); CFI[CFA]: .990), complying with the standards of goodness-of-fit. The loadings on the general factor ranged from small to high (Table 5), the highest being in the simple loop factor (mean =.74), followed by the sequence factor (mean =.58), nested loop factor (mean =.51), and conditional factor (mean =.38). In other words, the pattern of loadings resembled those of the discrimination parameter estimates in the bi-factor MIRT model, with higher loadings on the specific factors for more difficult items and higher loadings on the general factor for easier items. The reliability of the general factor was adequate (ωH= .79), indicating that nearly 80% of the test variance could be summarized in a single general factor score. On the other hand, the loadings on the specific factors were small- to-medium, presenting higher values in the items from the conditional block, and higher in the nested loop block than in the simple loop block. The BFM revealed some candidate items for removal. The weakest-defined specific factor was the simple loop factor, where items 7 and 8 had loadings close to zero (i.e.,. 08 and .01, respectively) and items 9 and 11 displayed low loadings (i.e., .27 an d .28, respectively). Item 20 also loaded weakly on the conditional factor (i.e.,.28). All the remaining items presented medium or high loadings on their respective specific factors. With regards to the general factor, all the items in the sequence, simple loop, and nested loop blocks had either medium or high loadings on the general factor, whereas most items from the conditional block displayed medium loadings. Only item 24 had a low loading on the general factor (i.e., .23). Next, we looked at the MI of the BFM to identify possible sources of model misspecification. The MI indicated that the BFM could be significantly improved if the nested loop factor was allowed to correlate with the conditional factor (MI = 52.21). Similarly, the sequence factor could be correlated with both the loop nested factor (MI = 29.20) and the simple loop factor (MI = 20.70), whereas the conditional factor could be correlated with both the simple loop factor (MI = 9.10) and the sequence factor (MI = 7.64). The only factor correlation that could not significantly improve the fit was that between the simple loop and nested loop factors. As it was shown before, withdrawing the test became more common from the nested loop block onward, so residual dependencies between the nested loop and conditional factors could be explained by the presence of these many absent responses. On the other hand, if the correlation between the sequence and nested loop factors was estimated, then it would be negative, which is compatible with the fact that some low-performing children withdrew when arriving at the nested loop block. With regards to cross-loadings, the MIs indicated that estimating many of them could also significantly improve the fit of the bi-factor model. However, we could not figure out an explanation for them (e.g., allowing item 14 to load on the conditional factor would lead to a fit improvement of MI = 18.41, but item 14 does not contain any conditional instruction). Finally, we found residual correlations between items 20 and 18 (MI = 11.92), 19 and 15 (MI = 9.57), 20 and 5 (MI = 7.30), and 23 and 19 (MI = 6.86). After inspecting all these item pairs in search for similarities, we did not found any apparent relationship in terms of content or difficulty except for items 23 and 19. Both of these were much more easier than the remaining items from the conditional factor, possibly because the response alternatives were bad and children could spot the correct answer by guessing. To confirm the adequate psychometric properties of this final 12-item selection, we refitted the BFM (Table 7). The BFM displayed again excellent absolute and relative fit indices (uSRMR: .51 (95%CI: .38–.65); RMSEA: .037 (95% CI: .022–.051, CFI: 0.986)) and the reliability of the general factor was also good (ωH=.80). Regarding the loadings, only items 9 and 11 had low loadings on their specific factor (i.e., values of .11 and .14 on the simple loop factor, respectively), whereas the remaining item loadings on the general and specific factors were medium or high. Bi-Factor MIRT Model An important detail is that missing responses should not have a parameter accounting for guessing because in these cases, 123 Journal of Science Education and Technology (2024) 33:998–1020 1017 Table 7 Factor loading estimates and 95% confidence intervals (within parentheses) for the bi-factor model of the final 12-item selection (BCTt-SF) Item Factor General factor loading Specific factor loading 3 Sequence 0.62 (0.52, 0.72) 0.45 (0.25, 0.65) 4 0.57 (0.47, 0.67) 0.46 (0.26, 0.66) 5 0.56 (0.46, 0.66) 0.45 (0.25, 0.65) 9 Simple loop 0.80 (0.71, 0.89) 0.11 (−0.33,0.55) 10 0.85 (0.76, 0.94) 0.52 (−1.14,2.17) 11 0.61 (0.50, 0.71) 0.14 (−0.41,0.68) 14 Nested loop 0.42 (0.31, 0.52) 0.50 (0.25, 0.76) 16 0.58 (0.48, 0.67) 0.42 (0.20, 0.64) 18 0.52 (0.42, 0.63) 0.34 (0.14, 0.53) 20 Conditional 0.50 (0.39, 0.61) 0.35 (0.18, 0.51) 21 0.42 (0.31, 0.53) 0.69 (0.43, 0.95) 22 0.38 (0.26, 0.50) 0.49 (0.28, 0.70) the items were never tried, rendering the probability of succeeding the item to be zero. Furthermore, as many of the missing responses happened because the examinees withdrew the test, we impeded missing values to contribute to the likelihood in the bayesian IRT model. Let be a parametervector collecting all the parameters of the model (1), so the (Bernoulli) probability mass function of a single observation can be written as P(Y=y|)=⎧ ⎪ ⎨ ⎪ ⎩ p,if y=1; 1−p,if y=0; 0,if yis missing. The posterior distribution of the parameters is then proportional to P(|y)∝P(y|)P(), where P()is the prior probability of the parameters, a quantity reflecting our epistemic knowledge before seeing the data. We defined priors on all the item discrimination parameters (i.e., αjand αgj) using a half-normal distribution with a location hyperparameter for each factor and one scale hyperparameter. Following the advice of Bürkner (2021)of using a distribution with mode at zero and then decreasing the probability density for larger values. The discrimination hyperparameters had another half-normal distribution centered at zero and with standard deviation one. We also chose the same half-normal prior for the standard deviation of the location parameters (dj) but a standard normal prior over the means for each factor. To resemble our uncertainty about the guessing parameters (cj), we chose a beta distribution with hyperparameters a=1 and b=3, meaning that from 4 trials, we expected one success and three failures if the examinee answers the item without knowing the correct alternative. With this distribution, we a priori expected that all the alternatives were equally attractive. Finally, we assigned a standard normal prior distribution to the new ηjparameters. Finally, we fixed the metric of the latent factors with a normal prior distribution with location zero and standard deviation one. All the factors were assumed to be independently distributed. Overall, the full model configuration was μαg,μ αs,σ α,σ d∼Normal(0,1), (μαg,μ αs,σ α,σ d)≥0, μds∼Normal(0,1), αj∼Normal(μαs,σ α), αj≥0, αgj ∼Normal(μαg,σ α), αgj ≥0, dj∼Normal(μds,σ d), ηj∼Normal(0,1), cj∼Beta(1,3), (θg,θ s)∼Normal(0,1). For estimating the bayesian model (1), we run five chains with a Hamiltoniam Monte Carlo algorithm and extracted 22,000 samples from the posterior distribution in each chain, discarding the first 2000 iterations as warmup. In total, 100,000 samples were extracted from the joint posterior distribution. The model setup, posterior convergence, and results were validated following the statistical workflow in Gelman et al. (2020). For instance, we conducted predictive prior checks to ensure that the prior distributions produced meaningful data. More concretely, we generated a prior predictive distribution of the total score. To account for the missing responses, we inserted zeros in the same positions of the simulated datasets than in the real dataset. Figure S1 illustrates this prior predictive check. The mean of the distribution was 10.80, with an standard deviation of 3.58 and interquartile range of [8.70−13.27]. We deemed these statistics to be a reasonable representation of realistic data. We also inspected the sampling diagnostics of the posterior distributions of the parameters. For assessing the 1018 Journal of Science Education and Technology (2024) 33:998–1020 convergence of the samples to the intended posterior distribution, we computed the Rhat statistic, the bulk effective sample size (ESSbulk), and the tail effective sample size (ESStail). Rhat is a measure of how well the Markov chains mixed. Typically, Rhat values less than 1.05 are considered to be good indicators that the Markov chains converged to the same posterior distribution. On the other hand, ESSbulk is a measure of sampling efficiency for the EAP estimate of the posterior distribution, whereas ESStail is the minimum effective sample size for 5% and 95% of the posterior quantiles. Both ESSbulk and ESStail should be at least 100 per Markov chain to have reliable estimates of the posterior quantiles. Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s10956-024-10142- 8. Funding This research was supported by Grant 22.167L (Ministry of Education and Vocational Training, Spain). Project name: The study and analysis of the benefits in the learning and development of early childhood education and 1st, 2nd, and 3rd grade primary education students through the elaboration and implementation of unplugged activities (without a computer). This study was also supported by the INVESTIGO program within the Recovery, Transformation and Resilience Plan, funded by the European Union (Next Generation EU project). This work is also co-funded by the Erasmus+ project CoTEDI, which is also co-financed by the European Union under the call-key action 2023-1-NL01-KA220-SCH-000152037 - OID E10207981. The work of J. Moreno-León was partially supported by FEDER/Ministry of Science, Innovation and Universities/Junta de Andalucía/State Research Agency/CDTI with the following grants: Data-pl (PID2022-138486OB-I00), TASOVA PLUS research network (RED2022-134337-T), MIDAS (IDI-20230256) and AquaIA (GOPGSE-23-0011). Data Availability The dataset used in this study and all the files necessary to reproduce the analyses are available at https://osf.io/3xn7a/? view_only=7190e194469f44119aa9174b768a406d. Declarations Conflict of Interest The authors declare no competing interests. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. References Aho, A. V. (2012). Computation and computational thinking. The Computer Journal, 55(7), 832–835. https://doi.org/10.1093/comjnl/ bxs074 Andrews, G., Halford, G., & Boyce, J. (2012). Conditional discrimination in young children: The roles of associative and relational processing. Journal of Experimental Child Psychology, 112, 84– 101. https://doi.org/10.1016/j.jecp.2011.12.004 Angeli, C., & Valanides, N. (2020). Developing young children’s computational thinking with educational robotics: An interaction effect between gender and scaffolding strategy. Computers in Human Behavior, 105, 105954. https://doi.org/10.1016/j.chb. 2019.03.018 Bakala, E., Gerosa, A., Hourcade, J., & Tejera, G. (2021). Preschool children, robots, and computational thinking: A systematic review. International Journal of Child-Computer Interaction, 29, 100337. https://doi.org/10.1016/j.ijcci.2021.100337 Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. https://doi.org/10.1037/0033- 2909.107.2.238 Bers, M., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. Computers & Education, 72, 145– 157. https://doi.org/10.1016/j.compedu.2013.10.020 Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., & Engelhardt, K. (2016). Developing computational thinking in compulsory education - Implications for policy and practice. In JRC Publications Repository.https://doi.org/10.2791/792158 Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Bürkner, P. -C. (2021). Bayesian item response modeling in R with brms and Stan. Journal of Statistical Software, 100.https://doi.org/10. 18637/jss.v100.i05 Caballero-Gonzalez, Y., & García-Valcárcel, A. (2018). A roboticsbased approach to foster programming skills and computational thinking: Pilot experience in the classroom of early childhood education. 41–45. https://doi.org/10.1145/3284179.3284188 Cai, L., Yang, J. S., & Hansen, M. (2011). Generalized full-information item bifactor analysis. Psychological Methods, 16, 221–248. https://doi.org/10.1037/a0023350 Diamond, A., Carlson, S., & Beck, D. (2005). Preschool children’s performance in task switching on the dimensional change card sort task: Separating the dimensions aids the ability to switch. Developmental Neuropsychology, 28, 689–729. https://doi.org/10. 1207/s15326942dn2802_7 El-Hamamsy, L., Zapata-Cáceres, M., Marcelino, P., Bruno, B., Dehler Zufferey, J., Martín-Barroso, E., & Román-González, M. (2022a). Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners’ CT test (BCTt) and the competent CT test (cCTt). Frontiers in Psychology, 13.https://www.frontiersin.org/articles/ 10.3389/fpsyg.2022.1082659 El-Hamamsy, L., Zapata-Cáceres, M., Barroso, E. M., Mondada, F., Zufferey, J. D., & Bruno, B. (2022b). The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Computing Research, 60(7), 1818–1866. https://doi. org/10.1177/07356331221081753 Elkin, M., Sullivan, A., & Bers, M. (2016). Programming with the KIBO robotics kit in preschool classrooms. Computers in the Schools, 33, 169–186. https://doi.org/10.1080/07380569.2016.1216251 Friedman, J., Hastie, T., & Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics (Oxford, 123 Journal of Science Education and Technology (2024) 33:998–1020 1019 England), 9(3), 432–441. https://doi.org/10.1093/biostatistics/ kxm045 Gabry, J., & ˇ Cešnovar, R. (2021). Cmdstanr: R Interface to ’CmdStan’. Garcia-Garzon, E., Abad, F. J., & Garrido, L. E. (2019). Improving bifactor exploratory modeling. Methodology, 15(2), 45–55. https:// doi.org/10.1027/1614-2241/a000163 Gelman, A., Vehtari, A., Simpson, D., Margossian, C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., & Modrák, M. (2020). Bayesian workflow. Georgiou, K., & Angeli, C. (2019). Developing preschool children’s Computational Thinking with educational robotics: The role of cognitive differences and scaffolding. 101–108. https://doi.org/10. 33965/celda2019_201911L013 Golino, H., & Christensen, A. P. (2022). EGAnet: Exploratory graph analysis – A framework for estimating the number of dimensions in multivariate data using network psychometrics. Golino, H. F., & Demetriou, A. (2017a). Estimating the dimensionality of intelligence like data using Exploratory Graph Analysis. Intelligence, 62, 54–70. https://doi.org/10.1016/j.intell.2017.02.007 Golino, H., & Epskamp, S. (2017b). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035. https://doi.org/10. 1371/journal.pone.0174035 Guggemos, J., Seufert, S., & Román-González, M. (2022). Computational thinking assessment - Towards more vivid interpretations. Technology, Knowledge and Learning. https://doi.org/10.1007/ s10758-021-09587-2 Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118 Jiménez, M., Abad, F. J., Garcia-Garzon, E., Golino, H., Christensen, A. P., & Garrido, L. E. (2023). Dimensionality assessment in bifactor structures with multiple general factors: A network psychometrics approach. Psychological methods, 10.1037/met0000590. Advance online publication. https://doi.org/10.1037/met0000590 Kong, S.-C. (2016). A framework of curriculum design for computational thinking development in K-12 education. Journal of Computers in Education, 3(4), 377–394. https://doi.org/10.1007/ s40692-016-0076-z Ling, D. S., Wong, C. D., & Diamond, A. (2021). Children only 3 years old can succeed at conditional “If, Then” reasoning, much earlier than anyone had thought possible. Frontiers in Psychology, 11. https://www.frontiersin.org/articles/10.3389/fpsyg.2020.571891 Li, Y., Xu, S., & Liu, J. (2021). Development and validation of computational thinking assessment of Chinese elementary school students. Journal of Pacific Rim Psychology, 15, 18344909211010240. https://doi.org/10.1177/18344909211010240 MacCallum, R. C. (2003). Working with imperfect models. Multivariate Behavioral Research, 38(1), 113–139. https://doi.org/10.1207/ S15327906MBR3801_5 Manches, A., & Plowman, L. (2015). Computing education in children’s early years: A call for debate. British Journal of Educational Technology,48, n/a–n/a. https://doi.org/10.1111/bjet.12355 Maydeu-Olivares, A. (2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82(3), 533–558. https://doi. org/10.1007/s11336-016-9552-7 McCormick, K., & Hall, J. (2022). Computational thinking learning experiences, outcomes, and research in preschool settings: A scoping review of literature. Education and Information Technologies, 27, 1–36. https://doi.org/10.1007/s10639-021-10765-z McDonald, R., & Ho, M.-H. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7, 64–82. https://doi.org/10.1037/1082-989X.7.1.64 Nishida, T., Kanemune, S., Idosaka, Y., Namiki, M., Bell, T., & Kuno, Y. (2009). A CS unplugged design pattern. ACM SIGCSE Bulletin, 41(1), 231–235. https://doi.org/10.1145/1539024.1508951 Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. https:// doi.org/10.1080/00273171.2012.715555 Relkin, E., & Bers, M. (2021). TechCheck-K: A Measure of computational thinking for kindergarten children. 2021 IEEE Global Engineering Education Conference (EDUCON), 1696–1702. https:// doi.org/10.1109/EDUCON46332.2021.9453926 Relkin, E., Johnson, S. K., & Bers, M. U. (2023). A normative analysis of the TechCheck computational thinking assessment. Educational Technology & Society, 26(2), 118–130. https://www.jstor. org/stable/48721000 Relkin, E., de Ruiter, L., & Bers, M. U. (2020). TechCheck: Development and validation of an unplugged assessment of computational thinking in early childhood education. Journal of Science Education and Technology, 29(4), 482–498. https://doi.org/10.1007/ s10956-020-09831-x Rodriguez, A., Reise, S., & Haviland, M. (2015). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods,21,. https://doi.org/10.1037/met0000045 Román-González, M., Moreno-León, J., & Robles, G. (2019). Combining assessment tools for a comprehensive evaluation of Computational Thinking interventions. In S.-C. Kong & H. Abelson (Eds.), Computational thinking education (pp. 79–98). Springer. https:// doi.org/10.1007/978-981-13-6528-7_6 Román-González, M., Pérez-González, J.-C., & Jiménez-Fernández, C. (2015). Test de Pensamiento Computacional: Diseño y psicometría general [Computational Thinking Test: Design & general psychometry].https://doi.org/10.13140/RG.2.1.3056.5521 Román-González, M., Pérez-González, J.-C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the computational thinking test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb. 2016.08.047 Román-González, M., Pérez-González, J.-C., Moreno-León, J., & Robles, G. (2018). Can computational talent be detected? Predictive validity of the Computational Thinking Test. International Journal of Child-Computer Interaction, 18, 47–58. https://doi.org/10. 1016/j.ijcci.2018.06.004 Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/ 10.18637/jss.v048.i02 Saxena, A., Lo, C. K., Hew, K., & Wong, G. (2020). Designing unplugged and plugged activities to cultivate computational thinking: An exploratory study in early childhood education. The Asia-Pacific Education Researcher, 29, 55–66. https://doi.org/10. 1007/s40299-019-00478-w Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003 Stamatios, P. (2021). The impact of coding apps to support young children in computational chinking and computational fluency. A literature review. Frontiers in Education, 6.https://www.frontiersin. org/articles/10.3389/feduc.2021.657895 Stamatios, P. (2022). Can preschoolers learn computational thinking and coding skills with ScratchJr? A systematic literature review. International Journal of Educational Reform,10567879221076076,. https://doi.org/10.1177/10567879221076077 Steiger, J. H. (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduk and Glaser. Structural Equation Modeling: A Multidisciplinary Journal, 7(2), 149–162. https://doi.org/10.1207/ S15328007SEM0702_1 1020 Journal of Science Education and Technology (2024) 33:998–1020 Tang, X., Yin, Y., Lin, Q., Hadad, R., & Zhai, X. (2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j. compedu.2019.103798 Taylor, M. (2018). Computer programming with PreK-1st grade students with intellectual disabilities. The Journal of Special Education, 52, 002246691876112. https://doi.org/10.1177/ 0022466918761120 Touretzky, D., Martin, F., Seehorn, D., Breazeal, C., & Posner, T. (2019). Special session: AI for K-12 guidelines initiative. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 492–493. https://doi.org/10.1145/3287324. 3287525 Tsarava, K., Moeller, K., Román-González, M., Golle, J., Leifheit, L., Butz, M. V., & Ninaus, M. (2022). A cognitive definition of computational thinking in primary education. Computers & Education, 179, 104425. https://doi.org/10.1016/j.compedu.2021.104425 Vourletsis, I., & Politis, P. (2024). Greek translation, cultural adaptation, and psychometric validation of beginners computational thinking test (BCTt). Education and Information Technologies.https://doi. org/10.1007/s10639-024-12887-6 Wang, X. C., Choi, Y., Benson, K., Eggleston, C., & Weber, D. (2020). Teacher’s role in fostering preschoolers’ computational thinking: An exploratory case study. Early Education and Development, 32, 1–23. https://doi.org/10.1080/10409289.2020.1759012 Weintrop, D., Wise Rutstein, D., Bienkowski, M., & McGee, S. (2021). Assessing computational thinking: An overview of the field. Computer Science Education, 31(2), 113–116. https://doi.org/10. 1080/08993408.2021.1918380 Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215 Zapata-Cáceres, Maria, & Fanchamps, N. (2021). Using the Beginners Computational Thinking Test to measure development on computational concepts among preschoolers: 5th APSCE international conference on computational thinking and stem education 2021. Proceedings of the 5th APSCE International Computational Thinking and STEM in Education Conference 2021, 32–37. Zapata-Cáceres, M., Martín-Barroso, E., & Román-González, M. (2020). Computational thinking test for beginners: Design and content validation. 2020 IEEE Global Engineering Education Conference (EDUCON), 1905–1914. https://doi.org/10.1109/ EDUCON45650.2020.9125368 Zapata-Cáceres, M., Martín-Barroso, E., & Román-González, M. (2021). BCTt: Beginners Computational Thinking Test. In Understanding computing education (Vol. 1, pp. 46–56). Raspberry Pi Foundation. Zhang, S., & Wong, G. K. W. (2023). Development and validation of a computational thinking test for lower primary school students. Educational Technology Research and Development.https://doi. org/10.1007/s11423-023-10231-2 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.