Patterns of action transitions in online collaborative problem solving : A network analysis approach

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Patterns of action transitions in online collaborative problem solving : A network analysis approach © The Author(s) 2022 Published version Li, Shupin; Pöysä-Tarhonen, Johanna; Häkkinen, Päivi Li, S., Pöysä-Tarhonen, J., & Häkkinen, P. (2022). Patterns of action transitions in online collaborative problem solving : A network analysis approach. International Journal of ComputerSupported Collaborative Learning, 17(2), 191-223. https://doi.org/10.1007/s11412-022-09369-7 2022 Received: 9 April 2021 / Accepted: 27 January 2022 © The Author(s) 2022 Extended author information available on the last page of the article Patterns of action transitions in online collaborative problem solving: A network analysis approach ShupinLi1· JohannaPöysä-Tarhonen1· PäiviHäkkinen1 Intern. J. Comput.-Support. Collab. Learn https://doi.org/10.1007/s11412-022-09369-7 1 3 Abstract In today’s digital society, computer-supported collaborative learning (CSCL) and collaborative problem solving (CPS) have received increasing attention. CPS studies have often emphasized outcomes such as skill levels of CPS, whereas the action transitions in the paths to solve the problems related to these outcomes have been scarcely studied. The patterns within action transitions are able to capture the mutual influence of actions conducted by pairs and demonstrate the productivity of students’ CPS. The purpose of the study presented in this paper is to examine Finnish sixth graders’ (N = 166) patterns of action transitions during CPS in a computer-based assessment environment in which the students worked in pairs. We also investigated the relation between patterns of action transitions and students’ social and cognitive skill levels related to CPS. The actions in the sequential processes of computer-based CPS tasks included using a mouse to drag objects and typing texts in chat windows. Applying social network analysis to the log file data generated from the assessment environment, we created transition networks using weighted directed networks (nodes for those actions conducted by paired students and directed links for the transitions between two actions when the first action is followed by the second action in sequence). To represent various patterns of action transitions in each transition network, we calculated the numbers of nodes (numbers of actions conducted), density (average frequency of transitions among actions), degree centralization (the dispersion of attempts given to different actions), reciprocity (the extent to which pairs revisit the previous one action immediately), and numbers of triadic patterns (numbers of different repeating formats within three actions). The results showed that pairs having at least one member with high social and high cognitive CPS skills conducted more actions and demonstrated a higher average frequency of action transitions with a higher tendency to conduct actions for different number of times, implying that they attempted more paths to solve the problem than the other pairs. This could be interpreted as the pairs having at least one student with high social and high cognitive CPS skills exhibiting more productive CPS than the other pairs. However, we did not find a significant difference across the pairs in terms of alternating sequences of two or three actions. Investigating the patterns of action transitions of the dyads in this study deepens our understanding of the mutual influence between the CPS actions occurring within dyads. Regarding pedagogical implication, our results offer empirical evidence recommending greater awareness of the students’ so- S. Li et al. 1 3 cial and cognitive capacities in CPS when assigning them into pairs for computer-based CPS tasks. Further, this study contributes to the methodological development of processoriented research in CSCL by integrating an analysis of action transition patterns with a skill-based assessment of CPS. Keywords Computer-supported collaborative learning · Collaborative problem solving · Collaborative problem solving skills · Action transitions · Social network analysis · Sixth graders Introduction In today’s knowledge-intensive and digitalized society, there is a critical need for learners to combine their expertise and ideas in various collaborative situations to solve problems together, both in individuals’ working lives and in a variety of learning communities. This issue has been addressed by research on computer-supported collaborative learning (CSCL). In CSCL, students share and construct knowledge with their group members through various interactions. In this way, we may consider collaboration as connections between diverse types of entities, such as students, actions, and the digital artifacts that the students create (van Aalst, 2009). Relations that are formulated within and/or between these entities are associated with effects on CSCL outcomes (Ouyang, 2021), embodying the philosophy of CSCL, which states that “relationships matter” (Dado & Bodemer, 2017, p.161). The present study investigated sequential actions in CSCL, including, for instance, clicking buttons, dragging to move objects with a mouse, and typing texts in the chat window. Particularly, we studied action transitions to better understand the relations within the sequential progressions of actions in dyadic interaction in an online assessment environment. The progressions of actions that a pair of students enact contribute towards processes of understanding, planning, solving, and revising, which are universally applicable across tasks in computer-based assessment environments (Care et al., 2015). This conceptualization provides a solid foundation for studying the processes of online assessment tasks through the lens of action sequences in the broader context of students’ efforts towards collaboration. The actions from two paired participants are interdependent, and their contributions “mutually build upon each other” (Hesse et al., 2015, p. 38). Therefore, the transition patterns between actions in sequences might capture this mutual influence between students in a CSCL environment. Today’s technologies for CSCL are becoming more advanced, and large amounts of computer-generated data are available, such as in the form of log files (Jeong et al., 2014). In addition to communication data (e.g., the contents of messages), log file data also record the sequences of actions that students conduct in CSCL environments. Contributing to inter-objective theories referencing networks of students, actions, and artifacts (Stahl & Hakkarainen, 2021), social network analyses have been widely applied to analyze the relations between entities in CSCL research (Dado & Bodemer, 2017). In the current study, applying social network analysis allowed us to capture the relations between the sequential actions contributed by each student in a dyad setting, here represented by patterns of action transitions. Patterns of action transitions in online collaborative problem solving: A… 1 3 Taking the form of short-term collaboration in CSCL (Reimann, 2009), collaborative problem solving (CPS) has been regarded as an important skill in the twenty-first century (Griffin et al., 2012; Ludvigsen et al., 2015; Rummel & Spada, 2005). CPS refers to collaboration as “a coordinated, synchronous activity that is the result of a continued attempt to construct and maintain a shared conception of a problem” (Roschelle & Teasley, 1995, p. 70). The social and cognitive skills of CPS have typically been studied in assessment research (Hesse et al., 2015). Administering CPS tasks designed for dyads in a computerbased assessment environment among sixth graders, we treated students’ social and cognitive skills of CPS as outcomes and worked to disentangle the relations between such outcomes and the patterns of action transitions contributed by pairs of students working together. This study deepens our understanding of the mutual influence of paired students’ actions related to their CPS skills in a computer-based CPS assessment environment. Moreover, our study contributes to the methodological development of process-oriented investigations in CSCL by integrating an analysis of action transition patterns with a skill-based assessment of CPS. Background Peer interactions and patterns of action transitions CSCL inherently involves intensive interaction between peers, which can be very effective in promoting learning (for a meta-analysis, see Tenenbaum et al., 2020). Peer interactions in CSCL are often mutual and untutored, involving “the use of small groups of students working together to achieve common goals of learning” (Topping et al., 2017, p. 5). In CSCL, peer interactions occur through the contingent, and therefore are in inherently sequential order of operational actions (e.g., moving objects with a mouse) and written/oral communication (Stahl & Hakkarainen, 2021). A given operational action or written/oral utterance is typically a response to the previous operational action or discourse move and is “generally designed to provoke a response and to propel the discourse and inquiry forward” (Stahl & Hakkarainen, 2021, p. 37). Intrinsically, contributions of these operational actions and written/oral communication exhibit mutual influence (Baker et al., 2007). Mutual influence between actions operates through the students who produce them. The phenomenon of mutual influence between actions is a matter of degree of strength, and it often operates within the sequentiality of actions (Stahl & Hakkarainen, 2021). Therefore, the patterns of action transitions (i.e., the transition between two actions with the first action followed by the second action, Zhu et al., 2016) might contribute towards the degree of mutual influence between actions. In the current article, by “actions,” we include operational actions and actions of writing messages for communication, though we exclude the specific content of messages. The interwoven progression of operational actions and written contributions leads to successful task solutions. However, the sequence of actions that will ultimately lead to a successful solution is not obvious to group members at the start of the task (Wieber et al., 2012), especially in a setting where individuals do not have identical resources in the online environment. Group members or pairs need to actively communicate with one another (e.g., type texts in the chat window) and to collectively attempt different operational actions (e.g., drag S. Li et al. 1 3 to move an object) to discover the path to the successful solution, often through trial-anderror. Accordingly, the action space affords high variability in possible action transitions. Based on U.S. data from PISA 2015, De Boeck & Scalise (2019) report that students who contribute more actions (e.g., click buttons, excluding actions of oral or written communication) are less successful. This may be because these PISA interactions are human–agent interactions (i.e., a student interacts with the computer-simulated agent rather than with another student; He et al., 2017), which results in limited opportunities for communication, which in turn may hinder demonstration of CPS skills when compared with the human– human approach (i.e., a student interacts with another student). In contrast to the findings of De Boeck & Scalise (2019), we assume that pairs conducting more actions in CPS tasks are likely to demonstrate better outcomes in the human–human approach because tasks in the human–human approach often require more actions, including those for the purpose of communication between partners. Another impetus for additional actions is that no structured assistance is offered, as it is for tasks implemented using the human–agent approach. Therefore, our study fills a research gap studying the relationship between the quality of students’ CPS outcomes and the number of actions conducted by paired students in the human–human approach, where the number of actions conducted may be very different than that in the prior work related to human–agent collaboration. On the other hand, Zhu et al. (2016) propose and validate two different patterns of action transitions in a computer-based problem solving task for individual students: First, they discuss the average frequency of action transitions (e.g., click available buttons, excluding actions of oral or written communication). In computer-based assessment environments, students need to attempt different paths to solve a problem. The more frequently students change actions on average as they engage in more exploration, the longer the total resulting sequence of actions. Second, they discuss the dispersion of attempts given to different actions. This pattern captures the extent to which students frequently conduct certain actions. The more dispersion of attempts given to different actions is present, the more students tend to conduct actions for the same number of times. In addition to studying the patterns of action transitions, Zhu et al. (2016) also relate patterns of action transitions to individual students’ problem solving outcomes. In particular, Zhu et al. (2016) investigate the action transitions in an online problem-solving task conducted by individual students whose efficiency scores associated with a scoring rubric are related to student outcomes in terms of their problem solving speed. In that case, speed is rated higher if students only take the necessary actions in order to reach a solution. The systematicity scores are also calculated that are related to how systematic students were at solving the problem. Here the systematicity score was rated higher if students follow the problem solving routine in the manual provided, meaning checking, fixing the problem, and testing the solution, respectively. Zhu et al. (2016) report that eighth graders with higher efficiency scores demonstrate a significantly lower average frequency of action transitions overall, whereas students with higher systematicity scores exhibit a more even distribution of probabilities across action sequences, indicating that they do not show a preference for certain action sequences over others. These findings are understandable given the conditions under which the study has been conducted in Zhu et al. (2016). That is, students in that study engaged in individual problem solving guided by a manual that provided the needed knowledge for the computer-based assessment task. In the work reported in this paper, in contrast, we focus on dyads engaging Patterns of action transitions in online collaborative problem solving: A… 1 3 in human–human collaboration, with no provided structured assistance related to the tasks provided for the participants (see details in Methods section). Consequently, we assumed that the frequency of action transitions, and dispersion of attempts, demonstrate the productivity of paired students’ CPS in computer-based assessment environments because these dimensions bound the space of possible measures of exploration pairs engage in while searching for a solution path. Compared with unproductive CPS, productive CPS is assumed in those dyads that conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times, a pattern that would stand in contrast to the findings in Zhu et al. (2016). In expanding from individual problem solving to problem solving in dyads, the present study fills a critical knowledge gap regarding how students’ CPS outcomes are related to their average frequency of shifts in action progressions and the dispersion of their attempts given to different actions, thus challenging the interpretation of past findings (Zhu et al., 2016). Moreover, Zhu et al. (2016) study the extent to which individual students revisit the most recent one or two actions during online assessment tasks. Information regarding the way students revisit these very recent actions can provide sophisticated insights into action design and the common pitfalls in problem-solving tasks (Zhu et al., 2016). For instance, if a certain action transition is frequently repeated yet is not a part of the path to the solution, this indicates that there might be a potential design problem in the task or that there is a common misconception hindering effective problem solving. Zhu et al. (2016) report that students who represent a higher level of systematicity scores (i.e., outcomes of individual students’ problem solving) to a lesser extent revisit the most recent previous actions right away (the repetition of the same actions over and over was not taken into account here). In our setting of dyadic human‒human collaboration, we build on past insights by exploring the extent to which dyads revisit the most recent one or two actions during a computer-based CPS task as an extension of existing knowledge about individual problem solving to dyadic CPS. CPS skills and group composition CPS has gained increasing societal interest, especially because of its inclusion in large-scale international assessments and in the development of computer-supported learning assessments (Shute & Rahimi, 2017). In 2015, the OECD integrated a CPS framework (OECD, 2017) into their PISA 2015 computerized assessments, utilizing human–agent tasks. Another important initiative to evaluate CPS is the Assessment and Teaching of 21st Century Skills (ATC21S) project (Care et al., 2018) that developed human–human computer-based assessment tasks in pairs. The theoretical framework of CPS in the present study relies on the one developed in the ATC21S project (Hesse et al., 2015). In this framework, a distinction is proposed between social and cognitive skills, which are further divided into a hierarchy of subskills (Hesse et al., 2015). This framework is not identical to the one in PISA 2015 that does not separate social skills from cognitive skills in CPS tasks. In Hesse et al.’s (2015) framework, social skills refer to the “collaborative” part of CPS that is often enacted through social interactions with the paired partner, including participation, perspective taking, and social regulation. Cognitive skills, on the other hand, refer to the “problem-solving” part of CPS, such as task regulation and knowledge building. The social and cognitive skills of CPS are inherently intertwined (Hesse et al., 2015). Establishing a mutual understanding requires S. Li et al. 1 3 members to socially engage in knowledge building and to constantly work to find common ground (Clark & Brennan, 1991), which calls for certain cognitive skills, such as collecting the elements of information related to the task. Small groups or pairs often experience major difficulties, particularly in establishing common frameworks of references, coming to a joint understanding, resolving differences of understanding, and negotiating individual and collective actions (Barron, 2000). Communication involving group members with high social skills is the key to resolving these challenges (Bause et al., 2018). Consequently, the skill composition of the group has been recognized as one contributor to the success of the collaborative learning process (Cen et al., 2016). The benefits of collaborative learning have often been argued as emerging through the assistance exchanged between collaborating partners (Stahl et al., 2006). For instance, with participants aged from 18 to 68, Dowell et al. (2020) report that groups with more members taking social responsibility performed better than those groups with greater proportions of less socially engaged partners. In school settings, students with a lower skill level can benefit from a collaborative learning situation more than students with a higher skill level (Saner et al., 1994). There are claims that poor group composition is one of the main reasons for unproductive collaborative learning (Fiechtner & Davis, 1985; Graf & Bekele, 2006). The optimal sharing of resources (e.g., existing skills, learning materials) has been suggested as vital for making collaborative learning more effective (Isotani et al., 2009). However, the reality in authentic classroom settings is that collaborating groups are often formed on a voluntary basis. In some cases, the voluntary selection of group members can result in off-task behaviors and resistance to group work (Dillenbourg, 2002). Therefore, there is a need for a retrospective investigation into the relationship between a group’s composition in terms of heterogeneous skill levels and the productivity of its collaboration. Mapping onto the PISA 2015 framework, Herborn et al. (2017) report that seventh graders’ individual profiles of social and cognitive skills differed in CPS performance in the human–agent approach. That is, students with high social and high cognitive skills demonstrated significantly better performance, whereas those with low social and low cognitive skills exhibited poorer performance. Based on Herborn et al. (2017), Andrews-Todd & Forsyth (2020) indicate that students with low social/low cognitive skill profiles exhibited the poorest performance in the human– human approach in triads. Moreover, having at least one high social and high cognitive member in a triadic group facilitates performance (Andrews-Todd & Forsyth, 2020). Herborn et al. (2017) and Andrews-Todd and Forsyth (2020), respectively, have uncovered a relationship between students’ performance related to individual skill profiles and the skill composition of groups with three members. Taking all this into account, the present study fills a research gap in the field by examining how students’ learning outcomes are related to their productive/unproductive CPS in human–human dyads. The quality of the solution for a CPS task has been the core criterion of interest that is traditionally regarded as the learning outcome (Graesser et al., 2017). However, log file data recorded in online CPS environments on students’ problem solving processes provide rich information on how students conduct various actions to solve a problem instead of merely whether the problem is solved or not (Zhu et al., 2016). In our study, we utilized CPS skill levels as learning outcomes calculated from log file data in the processes emerging from a series of CPS tasks. In line with the findings of Herborn et al. (2017) and Andrews-Todd & Forsyth (2020), we hypothesized that in computer-based CPS Patterns of action transitions in online collaborative problem solving: A… 1 3 assessment tasks, pairs having at least one member with high social and high cognitive skills might exhibit more productive CPS than other pair compositions. That is, dyads having at least one student with high social and high cognitive skills appear to conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times in CPS. Social network analysis (SNA) to study processes Alternative approaches exist for understanding different processes in computer-based learning environments, for instance, process-oriented content analysis (Isohätälä, 2020), process mining (Paans et al., 2019), and lag-sequential analysis (Malmberg et al., 2017). Other researchers have utilized Markov models and the item response theory framework (Shu et al., 2014) for analyzing process data. For modeling concurrent and sequential features in processes, Petri nets have also been utilized (Reisig, 1985). With their complex rules, it is claimed that Petri nets are not suitable for CSCL process data because “they are overly deterministic” (Reimann, 2009, p. 252). Therefore, heuristic methods that are more algorithmically complex and more readily to interpretate results need to be applied after Petri nets have been employed (Reimann, 2009). In addition to the methods mentioned above, SNA encompasses an important array of methods for analyzing various processes in CSCL as it allows for the analysis of patterns of relationships between two nodes interacting with one another in a relation-based system (Wasserman & Faust, 1994). In CSCL settings, nodes could be any entities in the CSCL processes: humans (e.g., instructors, students), artifacts (e.g., posts in a discussion forum), or types of online learning behaviors (e.g., content analysis with codes representing knowledge-building behaviors), depending on the specifics of target research questions (Dado & Bodemer, 2017). A link between two entities might connect entities with identical types (e.g., a link between students reveals a student communicates with the other one) or entities of different types (e.g., a link between a student and an activity reveals that the student participates in the activity). Admittedly, the definitions of nodes and of relational links play a vital role in the interpretations of analytical outcomes (Fincham et al., 2018). Some previous SNA research in CSCL has primarily focused on humans as nodes and communication between humans as links in the network (e.g., networks regarding communication-based interactions between students; Ouyang, 2021; Saqr et al., 2020). Some CSCL studies have employed SNA to address the temporality of discourses by utilizing qualitatively extracted parts of the participants’ discourse as nodes and interactions among the discourse of individuals as links (e.g., Swiecki et al., 2020; Zhang et al., 2021). On the other hand, Zhu et al. (2016) apply transition networks to analyze the process data generated from an online environment among individual students. They define actions (i.e., click a button) as the nodes and sequential transitions of actions as links to examine patterns of action transitions in individual students’ problem solving processes. The analysis of the previous research discussed above has relied on identical SNA theories, even though their research questions are tremendously different. Based on Zhu et al. (2016) and the fact that paired students’ sequential actions are intertwined and their contributions build one upon another (Hesse et al., 2015), we applied SNA methods to explore the patterns of action transitions in an online CPS assessment environment in a dyad setting with a human‒human approach. This fills a research gap in which S. Li et al. 1 3 there is scant research investigating paired students’ action transitions in computer-based CPS tasks. We created networks in which actions (e.g., drag to move an object, type messages) are nodes and the transitions of two actions are the links representing the order of the actions. We refer to these networks as transition networks (Zhu et al., 2016). Transition networks explicitly illustrate how students’ future actions are often influenced by their prior actions in CPS. Further, SNA measures are vital factors for assessing the formation and transition of entities (i.e., actions in our study) (Ouyang & Scharber, 2017), and SNA is often combined with other research methods to address the social and cognitive aspects of CSCL (Ouyang, 2021). Therefore, in addition to calculating SNA measures (see the Methods section) of transition networks as a way to represent patterns of action transitions, we also highlighted the distinct patterns of action transitions in CPS processes across differentiated pair compositions in terms of CPS social and cognitive skills. The process data from the present study are defined as a sequence of actions that the pairs conducted during CPS in an online assessment environment. Process data are typically represented by a sequence of actions, and each of these actions belongs to a finite pool of available actions (Zhu et al., 2016). In the ATC21S online assessment environment utilized in the present study, the action sequence was embedded with partial time information recorded by the environment in a way that the order of actions was relevant (i.e., the first action is followed by the second action in the sequence). Objectives The purpose of our study was to examine patterns of action transitions conducted by pairs of sixth-grade students in an online CPS assessment environment by utilizing an SNA approach. We also shed light on the relationship between the patterns of action transitions and students’ assessed skill levels of CPS (i.e., social and cognitive skills of CPS). Based on previous studies (e.g., Andrews-Todd & Forsyth, 2020), we hypothesize that pairs including at least one member with high social and high cognitive skills may conduct more actions with a higher frequency of action transitions on average and with a higher tendency to conduct actions for different number of times. The following research questions were addressed: 1. How do individual students’ social and cognitive skill levels vary in CPS tasks? What kinds of pair compositions can be identified in terms of the social and cognitive skills of CPS? 2. How do patterns of action transitions differ between pairs comprising diverse social and cognitive skill levels of CPS? Methods Participants and procedure With a convenience sampling method, data were collected in 2019 from 166 sixth-grade students (Meanage = 12.60, SDage = 0.33, female = 91, 54.82%) from 12 classes within 5 schools Patterns of action transitions in online collaborative problem solving: A… 1 3 of the degree centralization, the more unequal the degree values are. In the action transition networks from our process data, degree centralization measures the dispersion of attempts given to different actions. A low value of degree centralization shows that a pair conducts actions for the same number of times, while a high value of degree centralization indicates that a pair prefers to implement certain actions instead of others. Local dyadic and triadic measures. In addition to the global patterns in transition networks, we also examined local dyadic (two nodes) and triadic (three nodes) patterns because they constitute the basic blocks of the network structure and illustrate triadic dynamics at the local level. In the context of transition networks, these dyadic and triadic measures capture the local action transitions, which indicate the extent to which students revisited the previous action (i.e., dyadic pattern) and previous two actions (i.e., triadic patterns). Reciprocity (Wasserman & Faust, 1994) captures the dyadic structures in networks, and it is the number of mutual links divided by the total number of existing links. For transition networks generated from process data, reciprocity captures the extent to which the participants revisited the previous action right away. In the context of transition networks in the current article, it is worth noting that the meaning of reciprocity is different from that of the self-loop. Reciprocity exhibits the extent to which paired participants reconduct a previous different action, whereas self-loop refers to the previous same action repeatedly conducted (we excluded self-loops in the analysis as mentioned above). Triad census (Wasserman & Faust, 1994, p. 244) altogether has 16 link combinations made of three nodes. Figure 3 shows 16 possible triadic patterns that are named based on Holland & Leinhardt (1970) and Davis & Leinhardt (1972). The first digit in the names of the triads in Fig. 3 represents the number of reciprocal links in the triad; the second one reveals the number of nonreciprocal links in the triad; the third one shows the number of unconnected links in the triad; and a possible extra capitalized letter at the end represents the orientation of the triad in situations when the first three numbers are identical. The letter U means up; the letter D indicates down; the letter T reveals transitive; and the letter C represents cyclic. In networks generated from the process data, triadic structures show the extent to which the previous two actions tended to be revisited immediately. To obtain better insights into the transitive relationship among the three actions, we partitioned the 16 triadic patterns into five categories (see Table 3), here revised from Batagelj & Mrvar (2001) and Borgatti & Lopez-Kidwell (2014) in the context of transition networks that are generated from process data. Then, we calculated the numbers of 16 triadic patterns existing in each transition network for further analysis. All the global and local measures for transition networks we included (except the null triad that counts the number of isolated nodes) are confirmed to have significant predictive power related to the variability of students’ problem-solving outcomes in an online assessment environment (Zhu et al., 2016). Social and cognitive skills of CPS The social and cognitive skill levels of CPS were identified based on the log file data of all four tasks. The log file data consisted of mouse events (e.g., clicking a button, dragging to move an object) and chat discussion (i.e., typing texts in the chat window) in the task environment. All the actions were recorded in order and time-stamped. The focus of the ATC21S assessment tasks is “the process and quality of problem solving” (Adams et al., 2015, p. 116) rather than the conventional design that relies on attaining a solution as the sole cri- S. Li et al. 1 3 Table 3 Categories of triad census in transition networks Categories Triadic patterns (with names in Fig. 3) Descriptions Null triad 003 Three isolated nodes Dyadic triads 012, 102 Three nodes in which links exist between two nodes Brokerage triads 021D, 021U, 021C, 111D, 111U, 201 Partially connected three nodes with one node as a broker, capturing two links in a triad. Connected triads 030T, 030C, 120D, 120U, 120C, 210 Three non-reciprocally connected nodes, capturing three links in a triad. Reciprocal triad 300 Three reciprocally connected nodes Fig. 3 Patterns of triad census for directed networks. (Note: Nodes = actions that the participants conducted; Links between nodes = the sequences of action transitions; Arrows at the end(s) of links = directions of sequences in action transitions. A tie with an arrow at only one end between two nodes indicates that the participant changed from conducting one action to another but not the other way; a tie with an arrow at both ends between two nodes means that the participant alternated between a pair of actions.) Patterns of action transitions in online collaborative problem solving: A… 1 3 terion using dichotomous scores. Adopting rubrics and partial credit approaches, students’ social and cognitive skill levels of CPS were scored through automation procedures based on their actions in the log file during the problem solving for all four CPS tasks. The automation procedure began with the identification of task features matching the elements of Hesse et al.’s (2015) framework (i.e., participation, perspective taking, and social regulation represent social skills, while task regulation and knowledge building represent cognitive skills) from all the tasks administered. This was then followed by the generation of simple rules (see below) to collect data points that were able to represent these elements. The data points were extracted from log files generated by the students’ work in the assessment tasks, consisting of the documentation of each event (i.e., every action conducted by the participants). In particular, the actions observed in the log file data were used as indicators of social and cognitive skills, as defined in Hesse et al. (2015). Such indicative behaviors were then coded into rule-based indicators that could be extracted from the process log file data through an algorithmic procedure similar to the description in Zoanetti (2010), which reports how process data (e.g., action counts) can be interpreted as an indicator of a behavioral variable (e.g., learning from a mistake). These coded indicators were considered the primary data source for the scoring procedure. Each of the scoring algorithms took the coded dichotomous or polytomous indicators as the input and created a corresponding output, defined by the rule for the relevant indicator. For instance, the algorithm would count the occurrences of the event “chat” in the log file data if capturing the number of interactions in a task, and the output for this indicator would be a numerical value representing the frequency of the chat (for more details on the algorithms, see Adams et al., 2015). Then, the indicators were analyzed using Rasch modeling (Rasch, 1960) with two dimensions (i.e., social and cognitive skill levels). The modeling procedure set the average task indicator difficulty to 0, and the difficulty of an indicator was presented as an estimate describing the students’ skill level based on the four tasks. Consequently, the students’ skill levels were identified as higher if they conducted more actions whose corresponding theoretical indicators were more difficult to construct. The students’ social and cognitive skill levels were identified through weighted likelihood estimate scores (WLE; i.e., the estimates on item range of difficulty as enacted by the participant, which was given a measure of item difficulty). Table 4 shows the WLE distribution on the different skill levels of CPS. In practice, the scoring engine, which is managed by the University of Melbourne, automatically coded and scored the log file data, producing WLE scores and skill levels for further analysis and for producing reports for teacher and student use. Analysis strategy The data analysis was conducted in R 4.0.2 (R Core Team, 2020). R packages sna (v2.6; Butts 2020), network (v1.16.1; Butts et al., 2020), and GGally (v2.0.0; Schloerke et al., 2020) were mainly applied to process, visualize, and analyze the transition networks with the process data from the Olive Oil task. In particular, we created a transition network for each pair of the participants’ action transitions. We calculated the number of existing nodes, density, Freeman degree centralization, reciprocity, and the numbers of the 16 patterns in the triad census (see Fig. 3) for each network. Analysis of variance, Welch’s tests, and post hoc tests (Tukey and Bonferroni) were applied to the network measures across social and S. Li et al. 1 3 cognitive skill levels of CPS that were computed based on the four CPS tasks in the ATC21S portal. Results and discussion Social and cognitive skill levels of CPS among individuals and pair compositions Social and cognitive skills of CPS varied among the individual students in the four CPS tasks. The social skill levels of 153 (92.17%) participants were relatively high (i.e., at levels 4, 5, 6), whereas the cognitive skills of 120 (72.29%) participants were at the low levels (i.e., at levels 1, 2, 3). Table 5 shows the frequency of each social and cognitive skill level of CPS within the 166 individual participants. We categorized all participants into four theoretically based individual CPS profiles (Andrews-Todd & Forsyth, 2020). In particular, levels 1, 2, and 3 in social and cognitive skills (see Table 4) were grouped into the low dimension of skills, while levels 4, 5, and 6 were classified as the high dimension. In our data, there were three CPS groups for 166 individual participants: high social and high cognitive skills (HH), high social and low cognitive skills (HL), and low social and low cognitive skills (LL). Because the four CPS assessment tasks were administered in pairs, five pair compositions of CPS skills were identified among the 83 pairs of participants for further analysis. When the paired students exhibited the same levels of skills, we used “active” or “passive” to represent the high or low level of social skills, respectively, and used “high-performing” or “low-performing” to represent the high or low level of cognitive skills, respectively. “Compensated” was applied to demonstrate the pairs in which two students had different levels within the same skill (e.g., one member had high and the other had low social or cognitive skills). Accordingly, the five pair compositions of CPS skills were identified as active Skill levels Social skills: n (%) Cognitive skills: n (%) 1 4 (2.41%) 2 (1.21%) 2 6 (3.61%) 54 (32.53%) 3 3 (1.81%) 64 (38.55%) 4 35 (21.08%) 33 (19.88%) 5 94 (56.63%) 9 (5.42%) 6 24 (14.46%) 4 (2.41%) Table 5 Frequency of social and cognitive skill levels within 166 individual participants generated from a bundle of four CPS tasks Skill levels Social WLE range Cognitive WLE range 1 below 1.3 below − 3.5 2between − 1.3 and − 0.7 between − 3.5 and − 0.8 3between − 0.7 and − 0.5 between − 0.8 and 0.5 4between − 0.5 and 0.3 between 0.5 and 1.7 5 between 0.3 and 1.5 between 1.7 and 2.1 6 between 1.5 and 7 above 2.1 Table 4 Range of WLE scores in ATC21S portal corresponding to the social and cognitive skill levels of a bundle of four CPS tasks Patterns of action transitions in online collaborative problem solving: A… 1 3 high-performing pairs (both participants had high social and high cognitive skills), active compensated pairs (both participants had high social skills while each member represented either a high or low level of cognitive skills), active low-performing pairs (both participants demonstrated high social and low cognitive skills), compensated low-performing pairs (each member represented either high or low social skill levels while both exhibiting low cognitive skills), and passive low-performing pairs (both participants showed low social and low cognitive skills). Almost half of the pairs (n = 40, 48.19%) were active low-performing pairs, whereas passive low-performing pairs accounted for the fewest pairs (n = 2, 2.41%). It is notable that there was a set of 32 pairs in which each pair included at least one member with high social and high cognitive skills (i.e., active high-performing and active compensated pairs), whereas there were 51 pairs that did not have one member with high social and high cognitive skills (i.e., active low-performing, compensated low-performing, and passive low-performing pairs). Figure 4 presents the frequency distribution of the five pair compositions within the 83 pairs of participants. Utilizing a computer-based environment with a human−human approach, Andrews-Todd & Forsyth (2020) cluster four types of individual students’ CPS profiles of social and cognitive levels in a triad setting: high social and high cognitive, high social and low cognitive, low social and high cognitive, as well as low social and low cognitive. Except for the low social and high cognitive profile, we found the same profiles of individual students’ CPS skills as those in Andrews-Todd & Forsyth (2020). The reason why no individual students were assessed as having low social and high cognitive skills might be related to the asymmetric design of the Olive Oil task that deliberately promoted social interactions between paired partners. If dyads did not actively communicate with their partners through chatting, they were not able to proceed further in the task, leading to low instead of high cognitive skills of CPS in the assessment environment. Beyond the individual level, we extended the knowledge of individual students’ CPS profiles to five pair compositions in a dyad setting. Differences in patterns of action transitions among five pair compositions In this section, we first present and discuss visualizations of transition networks from action sequences within different pair compositions in order to highlight the action sequences indicative of skillful pairs in the context of the Olive Oil task. Second, we offer the descriptive statistics of network measures in these transition networks. Finally, we present and discuss comparisons across the network measures between transition networks associated with pairs with different skills. To illustrate the contrast in action transition processes across pair compositions, we visualized two transition networks, namely one conducted by an active high-performing pair and one by a passive low-performing pair; in both cases we chose the pairs who implemented the most actions among pairs that were similar in terms of composition of CPS skills. The comparison was based on the log file data of the Olive Oil task (see Figs. 5 and 6). The active high-performing pair consisted of two members with high social and high cognitive skills, whereas the passive low-performing pair had both members with low social and low cognitive skills. The active high-performing pair in Fig. 5 conducted more actions than the passive low-performing pair in Fig. 6 (i.e., there are more nodes in Fig. 5 than in Fig. 6). It can be seen that the active high-performing pair attempted a greater diversity of paths to solve the Olive Oil task. The active high-performing pair conducted 11 actions (see Fig. 5) S. Li et al. 1 3 out of the 12 necessary actions in the most efficient path (see 10 actions in Table 2 together with the nodes of ChatA and ChatB). In contrast, the passive low-performing pair implemented merely 4 actions (i.e., ChatA, ChatB, S1, and A7; see Fig. 6) that were in the most efficient path to the solution. On the other hand, the active high-performing pair repeated those actions that were not in the most efficient solution path less frequently than the passive low-performing pair (i.e., Fig. 6 has thicker blue edges revealing more frequent repetition between unnecessary actions in the most efficient path when compared with Fig. 5). Moreover, the active high-performing pair conducted a higher frequency of action transition when compared with the passive low-performing pair. Based on the above elaboration on the visualizations, Fig. 5 represents a relatively productive CPS, while Fig. 6 illustrates a relatively unproductive one. These visualizations of the transition networks show that the pair with high social and high cognitive skills of CPS appeared to conduct more productive Fig. 4 Frequency distribution of the five pair compositions (N = 83 pairs). (Note: HH-HH = active highperforming pairs (n = 14, 16.87%); HL-HH = active compensated pairs (n = 18, 21.69%); HL-HL = active low-performing pairs (n = 40, 48.19%); LL-HL = compensated low-performing pairs (n = 9, 10.84%); LL-LL = passive low-performing pairs (n = 2, 2.41%)) Patterns of action transitions in online collaborative problem solving: A… 1 3 CPS than the pair with lower skills. CSCL comprises processes of working collectively toward a solution of problems, especially when the path to the solution is unclear (Bause et al., 2018). Compared with unproductive CPS, productive CPS with more trials of different Fig. 5 Visualization of the transition network conducted by an active high-performing pair in the Olive Oil task. (Note: For meanings for the codes of the nodes, see Tables 1 and 2; Edge sizes equal to the weights of the edges divided by three. The reason for this division is to avoid an unclear presentation through visualization of overlapping edges; Colors of the edges: Red = action transitions among actions in the most efficient path to solve the problem; Blue = action transitions among actions not in the most efficient path to solve the problem; Gray = action transitions between actions necessary and not necessary in the most efficient path to solve the problem; The positions of nodes are randomly assigned, meaning that the distance between the nodes does not have particular meanings; this is decided randomly by the visualization software) Fig. 6 Visualization of the transition network conducted by a passive low-performing pair in the Olive Oil task, here following the same symbols as in Fig. 5 S. Li et al. 1 3 solution paths, and higher frequency of action transitions may better facilitate conducting the actions included in the most efficient path. In the transition networks generated from the process data in the Olive Oil task, the mean number of existing nodes was 27.8, meaning that 83 pairs conducted 27.8 actions on average during the task. The maximum number of actions that the pairs conducted was 58, while the minimum was 6, indicating a large variation between the pairs. The density of networks appeared to be sparse (M = 1.12%). This result implies that the participants were not likely to attempt every available action to work out the problem (some might have left the task before solving the problem). A low value in the mean of degree centralization (0.09) reveals that there appeared to be no particular focal action that the pairs implemented during the task. The low mean value of degree centralization is related to the asymmetric design of the Olive Oil task, leading to the solution path being unclear for paired students. Therefore, students had to communicate with their paired partners to attempt different paths to solve the problem. The mean of reciprocity was 0.24, indicating that the students revisited the previous one action immediately, at least to some extent. This implies that the students appeared to plan their solution paths with more than one action ahead. In terms of the triad census, the average number of null triads (i.e., three isolated nodes; see the “003” pattern in Fig. 3) turned out to be large (M = 71,881), meaning that there were 71,881 isolated “003” patterns on average. This result validates the low average value of density (1.12%) in the transition networks in which students did not appear to conduct many available actions. In turn, the average number of reciprocal triads (i.e., three nodes forming triadic reciprocity, see pattern “300” in Fig. 3) was small (M = 0.71). This shows that students barely revisited the previous two actions right away on average. That is, the student pairs were likely to attempt another path consisting of other actions to solve the problem rather than revisiting the previous two actions back and forth. Table 6 depicts the descriptive statistics of all network measures of the transition networks. In the current study, we utilized students’ social and cognitive skill levels of CPS as the outcome measures to guide formation of pairs based on composition. Based on the results of the ANOVA and Welch’s tests comparing network measures of transition networks across five pair compositions of CPS skills (see Table 7), all network measures were significantly different across pair compositions, except for reciprocity and the number of reciprocal triads (i.e., three actions forming triadic reciprocity, “300” in Fig. 3). The most important result is that the transition networks of pairs having at least one member with high social and high cognitive skills (i.e., active high-performing and active compensated pairs) demonstrated significantly higher mean values regarding the number of actions, average frequency of action transitions, and the tendency to conduct actions for different number of times when compared with the other pair compositions. First, pairs having at least one member with high social and high cognitive skills conducted more different actions than other pair compositions. This is further validated by our finding that the number of null triads (i.e., the number of three actions that were not conducted, e.g., “003” in Fig. 3) in the transition networks of active high-performing and active compensated pairs was significantly lower than that of other pairs (see Table 7). More concretely, the average number of actions that active high-performing pairs conducted was twice as much as that of compensated low-performing and passive low-performing pairs. Having at least one member with high social and high cognitive skills in a pair may facilitate the implementation of more actions to attempt different solution paths. In the Olive Oil Patterns of action transitions in online collaborative problem solving: A… 1 3 task, chats and all operational actions in the most efficient path of solution (see Table 2) are not easy to conduct successfully in just one trial. Pairs have to discuss and attempt different paths to access the solution. Pairs having at least one member with high social and high cognitive skills were more likely to obtain the potential solution for the task, leading to productive CPS because they conducted more different actions than other pairs when attempting different paths for the solution. On the other hand, our finding that pairs with the best skill levels (i.e., active highperforming pairs) conducted the most actions is not in line with the result in De Boeck & Scalise (2019), in which students who implemented more actions appeared to be less successful in PISA 2015. Compared with the human–human approach adopted in our study, the human–agent approach applied in De Boeck & Scalise (2019) provided students with limited opportunities for communication with the computer agent. Therefore, the path to solve the problem was designed to be relatively more structured so that the successful participants in the task did not necessarily need to conduct many actions to attempt different paths for the solution when compared with that in human–human approach tasks. Particularly, the Xandar task applied in De Boeck & Scalise (2019) has four parts (i.e., agreeing on a strategy, reaching a consensus, playing the game effectively, and assessing progress) that address a relatively clear path to solve the problem. For instance, in the first part of agreeing on a strategy, the student is expected to follow the rules of engagement provided, whereas in the fourth part, the agent poses a question about the progress. This structured form of assistance from the environment is useful for participants to figure out the solution path so that they Network measures Mean SD Max Min Number of existing nodes 27.80 13.60 58 6 Density 1.12% 0.56% 2.30% 0.12% Degree centralization 0.09 0.04 0.48 0.08 Reciprocity 0.24 0.08 0.19 0.02 Number of triad census: Null triad (003) 71881.00 2094.00 75627 67500 Dyadic triads 012 3403.00 1701.00 7040 370 102 517.00 255.00 1063 70 Brokerage triads 021D 44.00 40.20 185 1 021U 37.80 29.40 119 1 021C 97.90 78.20 350 1 111D 27.00 19.50 93 0 111U 26.90 22.10 92 1 201 3.43 3.84 17 0 Connected triads 030T 14.30 11.50 52 0 030C 5.63 4.56 18 0 120D 3.99 3.43 13 0 120U 4.60 3.73 17 0 120C 5.75 4.61 20 0 210 2.36 2.77 11 0 Reciprocal triad (300) 0.71 1.03 5 0 Table 6 Descriptive statistics of network measures in transition networks generated from the Olive Oil task S. Li et al. 1 3 do not need to conduct many actions to attempt different possible paths for the solution. In contrast, pairs representing the best skill levels were likely to conduct the most actions in the Olive Oil task with a human−human approach; this may be because of the asymmetric nature of the task, lack of structured assistance from the environment compared with the human–agent approach, and the restriction-free communication with partners. There could be substantial variations across computer-based CPS tasks in terms of the design of the tasks (e.g., whether the tasks are asymmetric or symmetric with a human–human or human–agent approach; how much and what kinds of the previous knowledge learned in the curriculum are required for the tasks; whether the communication is synchronous or not; whether the tasks are short-term, e.g., the tasks in our study, or long term that last for several weeks or months; or whether the tasks are for dyads, triads, or even more students in a group). To some extent, our results could be applied to those computer-based CPS tasks that are Table 7 Results of one-way ANOVA and Welch’s tests on five pair compositions across network measures in transition networks of process in the Olive Oil task Network measures Active highperforming (n = 14) (M/SD) Active compensated (n = 18) (M/SD) Active lowperforming (n = 40) (M/SD) Compensated low-performing (n = 9) (M/SD) Passive lowperforming (n = 2) (M/SD) F Number of nodes 37.79/13.17 32.72/14.14 25.48/11.31 15.89/9.56 13.50/6.36 6.38*** Density 1.40%/ 0.40% 1.40%/ 0.6% 1.00%/ 0.50% 0.50%/ 0.30% 0.60% /0.60% 7.29*** Degree centralization 0.12/0.04 0.11/0.04 0.08/0.03 0.05/0.01 0.06/0.04 11.52*** Reciprocity 0.24/0.08 0.25/0.05 0.24/0.07 0.26/0.14 0.21/0.08 0.18 Number of triad census: Null triad (003) 70813.64/ 1644.75 70723.17/ 2116.42 72190.98/ 1864.41 74049.33/ 1378.97 73799.00/ 1964.34 6.86*** Dyadic triads 012 4278.71/ 1404.94 4271.78/ 1693.48 3167.68/ 1525.32 1705.89 /1253.42 1851.50 /1495.53 6.21*** 102 612.00 /161.73 678.44 /253.22 484.05 /238.11 239.56 /132.91 304.00 /328.01 7.13*** Brokerage triads 021D 61.00/38.56 65.89/47.59 37.03/34.69 10.78/13.08 15.00/16.97 4.81** 021U 52.64/30.12 53.83/30.94 32.15/25.53 12.22/12.10 16.00/18.39 6.61*** 021C 141.50/79.70 142.78/89.07 80.60/62.16 29.78/31.67 40.00/39.60 6.37*** 111D 37.64/21.40 38.50/19.09 23.10/16.32 8.33/5.39 10.00/14.14 7.04*** 111U 35.86/23.73 41.83/24.95 22.35/17.14 7.67/6.96 6.00/5.66 10.95*** 201 4.14/3.35 5.44/4.62 3.00/3.66 0.78/1.39 1.00/1.41 3.03* Connected triads 030T 11.79/5.91 20.17/10.79 14.55/12.66 5.44/6.35 13.00/18.39 2.96* 030C 8.36/5.36 7.17/4.08 4.85/4.15 2.33/3.16 3.00/4.24 3.87** 120D 5.14/3.23 6.28/3.50 3.25/2.91 0.56/0.73 5.50/6.36 15.78*** 120U 5.43/2.93 7.11/4.09 4.10/3.28 0.56/0.53 4.50/6.36 21.73*** 120C 5.00/2.75 8.78/4.71 5.48/4.82 2.11/1.69 5.50/6.36 3.99** 210 2.21/2.16 4.11/3.64 2.08/2.44 0.56/1.33 1.50/2.12 3.25* Reciprocal triad (300) 0.93/1.00 0.72/0.83 0.78/1.21 0.11/0.33 0.50/0.71 0.98 Welch’s tests were applied to 111U, 120D, and 120U due to unequal variance, while a one-way ANOVA was applied to the rest of network measures Degree of freedom: (4, 78); *p < 0.05, **p < 0.01, ***p < 0.001 Patterns of action transitions in online collaborative problem solving: A… 1 3 Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–149). Washington, DC: American Psychological Association. https://doi.org/10.1037/10096-006 Dado, M., & Bodemer, D. (2017). A review of methodological applications of social network analysis in computer-supported collaborative learning. Educational Research Review, 22, 159–180. https://doi. org/10.1016/j.edurev.2017.08.005 Davis, J. A., & Leinhardt, S. (1972). The structure of positive interpersonal relations in small groups. In J. Berger (Ed.), Sociological theories in progress (2 vol., pp. 218–251). Boston, MA: Houghton Mifflin. De Boeck, P., & Scalise, K. (2019). Collaborative problem solving: Processing actions, time, and performance. Frontiers in Psychology, 10, 1280. https://doi.org/10.3389/fpsyg.2019.01280 Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In P. A. Kirschner (Ed.), Three worlds of CSCL. Can we support CSCL? (pp. 61–91). Heerlen: Open University Nederland. Dowell, N., Lin, Y., Godfrey, A., & Brooks, C. (2019, June). Promoting inclusivity through time-dynamic discourse analysis in digitally-mediated collaborative learning. In International Conference on Artificial Intelligence in Education (pp. 207–219). Cham: Springer. https://doi.org/10.1007/978-3-030-23204-7_18 Dowell, N., Lin, Y., Godfrey, A., & Brooks, C. (2020). Exploring the relationship between emergent sociocognitive roles, collaborative problem-solving skills, and outcomes: A group communication analysis. Journal of Learning Analytics, 7(1), 38–57. https://doi.org/10.18608/jla.2020.71.4 Fiechtner, S. B., & Davis, E. A. (1985). Why some groups fail: A survey of students’ experiences with learning groups. Organizational Behavior Teaching Review, 9(4), 75–88. https://doi. org/10.1177/105256298400900409 Fincham, E., Gašević, D., & Pardo, A. (2018). From social ties to network processes: Do tie definitions matter? Journal of Learning Analytics, 5(2), 9–28. https://doi.org/10.18608/jla.2018.52.2 Freeman, L. (1979). Centrality in social networks: Conceptual clarification. Social Networks, 1, 215–239. https://doi.org/10.1016/0378-8733(78)90021-7 Graesser, A., Kuo, B. C., & Liao, C. H. (2017). Complex problem solving in assessments of collaborative problem solving. Journal of Intelligence, 5(2), 10. https://doi.org/10.3390/jintelligence5020010 Graf, S., & Bekele, R. (2006). Forming heterogeneous groups for intelligent collaborative learning systems with ant colony optimization. In M. Ikeda, K. D. Ashley, & T-W. Chan (Eds.), Proceedings of Intelligent Tutoring Systems (ITS 2006), LNCS (Vol. 4053, pp. 217–226). Berlin, Heidelberg: Springer. https://doi. org/10.1007/11774303_22 Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. Dordrecht: Springer. https://doi.org/10.1007/978-94-017-9395-7 Griffin, P., McGaw, B., & Care, E. (Eds.). (2012). Assessment and teaching of 21st century skills. New York, NY: Springer. Hao, J., Liu, L., von Davier, A., & Kyllonen, P. (2015). Assessing collaborative problem solving with simulation based tasks. In O. Lindwall, P. Häkkinen, T. Koschmann, P. Tchounikine, & S. Ludvigsen (Eds.), Proceedings of the 11th International Conference on Computer Supported Collaborative Learning (pp. 544–547). Gothenburg, Sweden: International Society for the Learning Sciences. Retrieved from https://repository.isls.org/bitstream/1/462/1/297.pdf He, Q., von Davier, M., Greiff, S., Steinhauer, E. W., & Borysewicz, P. B. (2017). Collaborative problem solving measures in the Programme for International Student Assessment (PISA). In von A. Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 95–111). Dordrecht: Springer. https://doi.org/10.1007/978-3-319-33261-1_7 Herborn, K., Mustafić, M., & Greiff, S. (2017). Mapping an experiment-based assessment of collaborative behavior onto collaborative problem solving in PISA 2015: A cluster analysis approach for collaborator profiles. Journal of Educational Measurement, 54(1), 103–122. https://doi.org/10.1111/jedm.12135 Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin, & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 37–56). Dordrecht: Springer. https://doi.org/10.1007/978-94-017-9395-7_2 Holland, P. W., & Leinhardt, S. (1970). A method for detecting structure in sociometric data. American Journal of Sociology, 76, 492–513. https://doi.org/10.1016/B978-0-12-442450-0.50028-6 Isohätälä, J. (2020). The interplay of cognitive and socio-emotional processes in social interaction: Processoriented analyses of collaborative learning. Oulu: Acta Universitas Ouluensis. Isotani, S., Inaba, A., Ikeda, M., & Mizoguchi, R. (2009). An ontology engineering approach to the realization of theory-driven group formation. International Journal of Computer-Supported Collaborative Learning, 4(4), 445–478. https://doi.org/10.1007/s11412-009-9072-x Jeong, H., Hmelo-Silver, C. E., & Yu, Y. (2014). An examination of CSCL methodological practices and the influence of theoretical frameworks 2005–2009. International Journal of Computer-supported Collaborative Learning, 9(3), 305–334. https://doi.org/10.1007/s11412-014-9198-3 S. Li et al. 1 3 Johnson, D., Johnson, R., & Holubec, E. (1998). Cooperation in the classroom. Boston, MA: Allyn and Bacon. Liu, L., & Hmelo-Silver, C. E. (2010). Conceptual representation embodied in hypermedia: An approach to promoting knowledge co-construction. In M. S. Khine, & I. M. Saleh (Eds.), New science of learning: Cognition, computers and collaboration in education (pp. 341–356). New York, NY: Springer. https:// doi.org/10.1007/978-1-4419-5716-0_17 Liu, L., von Davier, A. A., Hao, J., Kyllonen, P., & Zapata-Rivera, J. D. (2015). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on computational tools for real-world skill development (pp. 344–359). Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-4666-9441-5.ch013 Ludvigsen, S. C. C., Gundersen, E., Kleven, K., Rege, M., Øye, H., Indregard, S. … Sundberg, D. (2015). The school of the future: Renewal of subjects and competences (Official Norwegian Reports NOU 2015: 8). Oslo: Norwegian Ministry of Education and Research. Malmberg, J., Järvelä, S., & Järvenoja, H. (2017). Capturing temporal and sequential patterns of self-, co-, and socially shared regulation in the context of collaborative learning. Contemporary Educational Psychology, 49, 160–174. https://doi.org/10.1016/j.cedpsych.2017.01.009 Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall. OECD (2017). Pisa 2015 collaborative problem-solving framework. Retrieved from https://www.oecd.org/ pisa/pisaproducts/pisa2015draftframeworks.htm Ouyang, F. (2021). Using three social network analysis approaches to understand computer-supported collaborative learning. Journal of Educational Computing Research, 735633121996477. https://doi. org/10.1177/0735633121996477 Ouyang, F., & Scharber, C. (2017). The influences of an experienced instructor’s discussion design and facilitation on an online learning community development: A social network analysis study. The Internet and Higher Education, 35, 34–47. https://doi.org/10.1016/j.iheduc.2017.07.002 Paans, C., Onan, E., Molenaar, I., Verhoeven, L., & Segers, E. (2019). How social challenges affect children’s regulation and assignment quality in hypermedia: A process mining study. Metacognition and Learning, 14(2), 189–213. https://doi.org/10.1007/s11409-019-09204-9 Poquet, O., Saqr, M., & Chen, B. (2021, April). Recommendations for network research in learning analytics: To open a conversation. In O. Poquet, B. Chen, M. Saqr, & T. Hecking (Eds.), Proceedings of the NetSciLA21 Workshop “Using network science in learning analytics: Building bridges towards a common agenda” (Issue 2868, pp. 34–41). Retrieved from http://ceur-ws.org/Vol-2868/article_7.pdf Pöysä-Tarhonen, J., Care, E., Awwal, N., & Häkkinen, P. (2018). Pair interactions in online assessments of collaborative problem solving: case-based portraits. Research and Practice in Technology Enhanced Learning, 13(1), 1–29. https://doi.org/10.1186/s41039-018-0079-7 R Core Team (2020). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/ Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Paedagogike Institut. Reimann, P. (2009). Time is precious: Variable-and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257. https:// doi.org/10.1007/s11412-009-9070-z Reisig, W. (1985). Petri Nets. An introduction. Berlin: Springer. Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.), Computer supported collaborative learning (pp. 69–97). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-85098-1_5 Rummel, N., & Spada, H. (2005). Learning to collaborate: An instructional approach to promoting collaborative problem solving in computer-mediated settings. The Journal of the Learning Sciences, 14(2), 201–241. https://doi.org/10.1207/s15327809jls1402_2 Saner, H., McCaffrey, D., Stecher, B., Klein, S., & Bell, R. (1994). The effects of working in pairs in science performance assessments. Educational Assessment, 2(4), 325–338. https://doi.org/10.1207/ s15326977ea0204_4 Saqr, M., Viberg, O., & Vartiainen, H. (2020). Capturing the participation and social dimensions of computersupported collaborative learning through social network analysis: Which method and measures matter? International Journal of Computer-Supported Collaborative Learning, 15(2), 227–248. https://doi. org/10.1007/s11412-020-09322-6 Schloerke, B., Cook, D., Larmarange, J., Briatte, F., Marbach, M., Thoen, E. … Crowley, J. (2020). GGally: Extension to ‘ggplot2’ (version 2.0.0) [R package]. Retrieved from https://CRAN.R-project.org/ package=GGally Scott, J., & Carrington, P. J. (2011). The SAGE handbook of social network analysis. Thousand Oaks, CA: SAGE. Patterns of action transitions in online collaborative problem solving: A… 1 3 Shu, Z., Zhu, M., Hao, J., Bergner, Y., & von Davier, A. A. (2014, July). Using Markov-IRT to characterize process data. Paper presented at the 79th annual meeting of the Psychometric Society, Madison, WI, USA Shute, V. J., & Rahimi, S. (2017). Review of computer-based assessment for learning in elementary and secondary education. Journal of Computer Assisted Learning, 33(1), 1–19. https://doi.org/10.1111/ jcal.12172 Stahl, G., & Hakkarainen, K. (2021). Theories of CSCL. In U. Cress, C. Rosé, A. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 23–44). Cham: Springer. https://doi.org/10.1007/978-3-030-65291-3_2 Stahl, G., Koschmann, T., & Suthers, D. (2006). CSCL: An historical perspective. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 409–426). Cambridge, UK: Cambridge University Press. Swiecki, Z., Ruis, A. R., Farrell, C., & Shaffer, D. W. (2020). Assessing individual contributions to collaborative problem solving: A network analysis approach. Computers in Human Behavior, 104, 105876. https:// doi.org/10.1016/j.chb.2019.01.009 Tenenbaum, H. R., Winstone, N. E., Leman, P. J., & Avery, R. E. (2020). How effective is peer interaction in facilitating learning? A meta-analysis. Journal of Educational Psychology, 112(7)), 1303–1319. https:// doi.org/10.1037/edu0000436 Theiner, G., & O’Connor, T. (2010). The emergence of group cognition. In A. Corradini, & T. O’Connor (Eds.), Emergence in science and philosophy (pp. 79–117). New York, NY: Routledge. Topping, K., Buchs, C., Duran, D., & Van Keer, H. (2017). Effective peer learning: From principles to practical implementation. London: Routledge. https://doi.org/10.4324/9781315695471 van Aalst, J. (2009). Distinguishing knowledge-sharing, knowledge-construction, and knowledge-creation discourses. International Journal of Computer-Supported Collaborative Learning, 4(3), 259–287. https:// doi.org/10.1007/s11412-009-9069-5 von Davier, A. A., & Halpin, P. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations. ETS Research Report Series, 2013(2), i–36. https://doi. org/10.1002/j.2333-8504.2013.tb02348.x Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge, UK: Cambridge University Press. Wieber, F., Thürmer, J. L., & Gollwitzer, P. M. (2012). Collective action control by goals and plans: Applying a self-regulation perspective to group performance. The American Journal of Psychology, 125(3), 275–290. https://doi.org/10.5406/amerjpsyc.125.3.0275 Wise, A. F., & Schwarz, B. B. (2017). Visions of CSCL: Eight provocations for the future of the field. International Journal of Computer-Supported Collaborative Learning, 12(4), 423–467. https://doi.org/10.1007/ s11412-017-9267-5 Zhang, S., Chen, J., Wen, Y., Chen, H., Gao, Q., & Wang, Q. (2021). Capturing regulatory patterns in online collaborative learning: A network analytic approach. International Journal of Computer-Supported Collaborative Learning, 16, 37–66. https://doi.org/10.1007/s11412-021-09339-5 Zhu, M., Shu, Z., & von Davier, A. A. (2016). Using networks to visualize and analyze process data for educational assessment. Journal of Educational Measurement, 53(2), 190–211. https://doi.org/10.1111/ jedm.12107 Zoanetti, N. P. (2010). Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology, 26(5), 585–606. https://doi. org/10.14742/ajet.1053 Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Authors and Affiliations ShupinLi1· JohannaPöysä-Tarhonen1· PäiviHäkkinen1 Shupin Li [email protected] 1 Finnish Institute for Educational Research, University of Jyväskylä, P.O. Box 35, FI-40014, Jyväskylä, Finland