Categorization of simulated diagnostic situations and the salience of diagnostic information

© 2023 The Author(s) Distributed as a Hogrefe OpenMind article Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) https://doi.org/10.1024/1010-0652/a000364 Original Article Categorization of simulated diagnostic situations and the salience of diagnostic information Conceptual framework Nils Machts1 , Olga Chernikova2, Thorben Jansen3 , Marc Weidenbusch4 , Frank Fischer2, and Jens Möller1 1 Institute for Psychology of Learning and Instruction (IPL), Kiel University, Germany 2 Chair of Education and Educational Psychology, University of Munich (LMU), Germany 3 Leibniz Institute for Science and Mathematics Education (IPN), Kiel University, Germany 4 University Hospital, University of Munich (LMU), Germany Abstract: Simulated diagnostic situations function as learning environments and research tools to deepen the understanding of diagnostic processes. In this conceptual paper, we present a framework that fulfils two purposes. First, we categorize simulations of diagnostic situations using six descriptive components: physical environment, social embeddedness, diagnostic tasks, activities, information, and decisions. Second, we discriminate the salience of diagnostic information into four facets: access, prominence, simplicity, and clarity. These facets determine the likelihood that participants in a simulation will perceive and process relevant diagnostic information toward correct or accurate diagnostic decisions. We use one simulation to explain these components and facets in more detail. With this systematic conceptualization, we intend to deepen the understanding of how and why simulations work in creating authentic diagnostic situations, measuring diagnostic competencies, and providing learning environments for the facilitation of these competencies in higher education. Keywords: diagnostic competencies, simulations, learning environments Kategorisierung von Simulationen diagnostischer Situationen und die Bedeutung diagnostischer Informationen: Konzeptioneller Rahmen Zusammenfassung: Simulationen diagnostischer Situationen dienen sowohl als Lernumgebung als auch als Forschungsinstrumente zur Vertiefung des Verständnisses von diagnostischen Prozessen. In diesem konzeptionellen Papier stellen wir einen Rahmen vor, der zwei Zwecke erfüllt. Erstens kategorisieren wir Simulationen diagnostischer Situationen anhand von sechs deskriptiven Komponenten: physische Umgebung, soziale Einbettung, diagnostische Aufgaben, Aktivitäten, Informationen und Entscheidungen. Zweitens unterscheiden wir die Salienz der diagnostischen Informationen in vier Facetten: Zugänglichkeit, Auffälligkeit, Einfachheit und Klarheit. Diese Facetten bestimmen die Wahrscheinlichkeit, dass die Teilnehmer_innen an einer Simulation relevante diagnostische Informationen wahrnehmen und verarbeiten, um korrekte oder genaue diagnostische Entscheidungen treffen zu können. Wir verwenden eine Beispielsimulation, um die Komponenten und Facetten näher zu erläutern. Mit dieser systematischen Konzeptualisierung wollen wir das Verständnis dafür vertiefen, wie und warum Simulationen bei der Schaffung authentischer diagnostischer Situationen, der Messung diagnostischer Kompetenzen und der Bereitstellung von Lernumgebungen für die Förderung dieser Kompetenzen in der Hochschulbildung funktionieren. Schlüsselwörter: Diagnostische Kompetenzen, Simulationen, Lernumgebungen Introduction Making professional decisions based on knowledge of other people's characteristics is a crucial activity in many professional domains, such as diagnosing patients in medicine, determining clients' needs in counseling, or identifying learners' misconceptions in teaching. These activities require a range of skills related to collecting, evaluating, and utilizing diagnostic information. Training people in these skills is one of the essential tasks of higher education (e. g., Chernikova, Heitzmann, Fink, et al., 2020). One method for facilitating their further development is the imhttps://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 4 N. Machts et al., Categorization of simulated diagnostic situations Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) plementation of simulation-based learning (e. g., Cook, 2014; Chernikova, Heitzmann, Stadler, et al., 2020; Gartmeier et al., 2015). Simulated diagnostic situations are frequently used to investigate diagnostic processes in different domains (Codreanu et al., 2020; Kaiser et al., 2013; Südkamp et al., 2008) and to evaluate the effectiveness of learning and teaching strategies (e. g., Chernikova, Heitzmann, Stadler, et al., 2020). However, there is no framework that allows for cross-domain comparisons of simulation features, the generalizability of the obtained results, and the transfer of good practices between domains. Therefore, we aim to develop such a framework. In doing so, we rely on (1) the similarities in scientific reasoning and argumentation that underlie the diagnostic processes in medical and teacher education (see Fischer etal., 2014) and (2) the effects of problem-solving on the development of diagnostic competences across domains (Chernikova, Heitzmann, Fink, et al., 2020). In spite of all the possible differences between the contexts of real-life diagnostic situations, simulated diagnostic situations share several features across domains. A core feature that makes simulations effective tools for research and education is that they offer a high level of design control in terms of the tasks given to participants, the available diagnostic activities, and the diagnostic information that participants receive. This control allows for researching and facilitating diagnostic competencies in particular diagnostic situations, such as situations with a lot of uncertainty, rarely occurring situations, or situations focusing on individual diagnostic activities. However, gains in the internal validity of such controlled conditions can also increase their artificiality, posing a threat to aspects of external validity (e. g., perceived utility value and authenticity) that play a significant role in the effectiveness of simulations (Chernikova et al., 2024). This makes it a delicate but crucial task for the designers of simulations to find a balance between experimental control and the authentic representation of actual situations. Studies that utilize simulated diagnostic situations employ different measures of diagnostic competencies, such as judgment accuracy (e. g., Kaiser et al., 2017), other scoring methods of diagnostic success (e. g., Klug et al., 2013; Kron et al., 2021), and report various measures of the training effects (Chernikova, Heitzmann, Stadler, et al., 2020). A closer look at specific applications of simulations also reveals differences in how designers realize control over elements of the simulation. For example, the authors may decide to relieve participants from a diagnostic activity, such as the evaluation of diagnostic information, by labeling relevant pieces of information, thus making it easier to arrive at correct diagnostic decisions. Alternatively, they might overload participants with distracting irrelevant information, requiring participants to actively select relevant information. Ultimately, simulations can be adapted to various purposes, but similar effect sizes can yield different meanings depending on the simulation design. Existing conceptual frameworks describe the nature of diagnostic activities, individual prerequisites, instruction, and context (Chernikova et al., 2022; Heitzmann et al., 2019; Loibl et al., 2020); however, they do not provide a systematic breakdown of differences between how various diagnostic situations are applied to simulations, especially in terms of the presentation of diagnostic information and the ways of processing them. Consequently, we hope to establish a conceptual framework to account for these differences so that we can systematically categorize simulated diagnostic situations, explain variances in their results, and ultimately provide a common language for the design principles of simulated diagnostic situations used to measure and facilitate diagnostic competencies. Our approach entails two steps. First, we establish an outer framework of the structural components of simulated diagnostic situations. Existing modeling approaches provide insights into the structural elements of diagnostic situations in real-life contexts (Fiske & Neuberg, 1990; Karst et al., 2017; Loibl et al., 2020). Our framework builds on these existing concepts; however, is clearly oriented toward what Loibl et al. (2020) call the external aspects of the diagnostic situation, as these aspects are modifiable through simulation design principles. Second, we establish a focus on diagnostic information, elaborating on why it represents the core component of a diagnostic situation as the basis for each diagnostic decision. In doing so, we utilize the psychological term salience – the property of stimuli to stand out and be likely to draw a perceivers' selective attention (Higgins, 1996) – to classify the presentation of relevant diagnostic information in a simulation. Generally, our approach implies that a higher salience of relevant information increases the likelihood of participants making correct or accurate diagnostic decisions (Chernikova et al., 2024). Then, with our concept established, we present an example of how the framework can be applied to the research design of an existing simulated diagnostic situation. Finally, we discuss our concept using a research agenda outline deduced from its application possibilities, including the interplay between control over the salience of information and the perceived authenticity of a simulation. Simulated diagnostic situations For this framework, we define diagnostic situations from a minimalist perspective. In other words, we aim to identify https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 N. Machts et al., Categorization of simulated diagnostic situations 5 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) the necessary and sufficient components of diagnostic situations across domains, as well as the additional components that contribute to the variability of such situations. We also build our framework on the premise of being able to recreate a diagnostic situation accurately in a simulation. Accordingly, in reference to the recently introduced DiaCoM model of diagnostic situations (Loibl et al., 2020), individual diagnosticians' characteristics are of second nature to the proposed general structure of a diagnostic situation. Otherwise, our approach shares structural similarities, whereby we shift the focus toward finding families of technical components that simulation designers exercise control over. We propose that the general structure of diagnostic situations must necessarily include – in the broadest sense – diagnosticians who (a) identify diagnostic information on diagnostic targets and (b) apply this information to ensuing diagnostic decisions. In addition to these necessary elements, most diagnostic situations include objectives for performing diagnostics and a diagnostic process that includes various diagnostic activities to accomplish these objectives. Finally, diagnostic situations unfold under specific physical conditions and are embedded in some form of social interaction or social relevance. In simulations of diagnostic situations, designers create representations of such situations using varying sets of tools and degrees of proximity to examples from real life (Grossmann et al., 2009). Two important research fields for simulated diagnostic situations in higher education aremedicine and education, which – despite obvious disparities – share the general structure of diagnostic situations (Heitzmann et al., 2019). A main difference among simulation designs concerns whether participants interact with simulated persons and derive information at least partly from this interaction, or whether they acquire access through documented information (Heitzmann et al., 2019). Both the medical and educational fields offer representations of these two general design approaches. For example, in the educational field, Kaiser et al. (2017) used simulated students in a simulated classroom. Participants in the role of a teacher applied a question-answer schema as a form of simplified instruction for gathering information on their students' lesson performance by interacting with them. In the medical field, Liaw and Huang (2013) used a patient simulator that presented respiratory and cardiovascular distress symptoms, and participants performed acute care management based on their interactions with the patient simulator. Alternatively, in another example from the educational field, Jansen et al.'s (2021) participants had access to multiple student texts and had to make various judgments on the quality of these texts by processing the documented information at different analytical levels. In a similar example from the medical field, Chamberland et al. (2015) presented participants with clinical cases of jaundice that included the patient's chief complaint; background information; and findings from history-taking, a physical examination, and laboratory tests. The participants then had to process the documents to make diagnostic decisions about the underlying diseases. In both basic simulation design principles, the close relationship between content and learning scenarios in higher education institutions and real-world professional situations helps support future professionals in acquiring practice-oriented professional knowledge, skills, and competences (Blömeke et al., 2015). Although systematic research has revealed the large positive effects of simulationbased learning in different domains of higher education (Cook et al., 2013; Theelen et al., 2019), analyses have also revealed large heterogeneity in the effects, indicating that simulation-based learning environments and the included instructional support measures differ in their effectiveness (Chernikova, Heitzmann, Stadler, et al., 2020). Attempts to explore different features of simulation-based environments – such as the use of technology, instructional support, and learners' prerequisites (Chernikova, Heitzmann, Stadler, et al., 2020) – still leave much of this variance unexplained. It is important to consider the extent to which simulations represent actual practice in terms of the learner's demands, the nature of the simulated situation and the environment, and the participants involved (Allen et al., 1991). Hamstra et al. (2014) also emphasized the importance of distinguishing between the physical resemblance and functional correspondence of a task or scenario. However, simulated environments also offer the opportunity to select and modify representations of reality (Grossman et al., 2009). As such, simulations allow for focusing on specific aspects of actual tasks and processes, thus enabling the investigation of specific effects under experimental control and the inclusion of targeted training interventions. Establishing a clear conceptualization of such modifications should thus allow for the pinpointing of the working mechanisms of simulations and the identification of which modifications are necessary and which may threaten a simulation's perceived utility value and authenticity. In what follows, we expand on Loibl et al.'s (2020) and Heitzmann et al.'s (2019) frameworks and propose the systematic categorization of simulated diagnostic situations across six components, emphasizing the component of diagnostic information. We further establish a concept of salience of diagnostic information and categorize four facets. This conceptualization reveals that different simulations provide different opportunities to investigate and facilitate the validity of diagnostic decisions in terms of participants gaining access to relevant information, focusing on relevant information, managing complex decision https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 6 N. Machts et al., Categorization of simulated diagnostic situations Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) structures, and dealing with ambiguous information. With our example, we show that our proposed components can be smoothly applied to diagnostic situations and that the presented diagnostic information can be classified comprehensively using the facets of salience. With this conceptual framework, we lay the groundwork for a better theoretical understanding of the heterogeneity of the effects of differently simulated diagnostic situations (see Chernikova, Heitzmann, Stadler, et al., 2020). The framework can also serve as a common reference and language to describe simulated diagnostic situations and, thus, systematize research. A better understanding of and more systematic research on simulations can ultimately help derive design principles for simulations of different professional diagnostic situations and for participants with different prerequisites. Conceptual framework Components of simulated diagnostic situations Although simulated diagnostic situations have fundamental differences in terms of their basic design principles, they also have clear parallels across different domains and applications (Chernikova, Heitzmann, Fink, et al., 2020). In this chapter, we outline six components that enable the systematic description and differentiation of simulated diagnostic situations (see Figure1). Beyond the categoriza1 For example, in a core diagnostic situation, a teacher may receive a student answer and the simulation may ask for an estimation regarding whether the student understood the topic. Neither the diagnostic task nor activity are required for this core situation. tion of different simulations, these components should also provide a clear visualization of where in a simulation designers have the opportunity to create variations and implement interventions to investigate and increase learning gains in measures of diagnostic success through the instructional application of such simulations. It is important to note that by focusing on the design choices in simulations, we exclude individual participants' characteristics and states from the components of the simulated diagnostic situation. This leads to a clear distinction between this framework of diagnostic situations and the DiaCoM model (Loibl et al., 2020). The general idea is as follows: A simulation of a diagnostic situation has a physical environment and is socially embedded both within and outside itself. The inner part represents the actual simulation and must always include the core components of diagnostic situations: diagnostic information and diagnostic decisions.1 Additionally, it usually includes diagnostic tasks and activities. The logical process of a simulation entails setting the task, starting the activities, ending with a pool of information, and concluding with a diagnostic decision. Nevertheless, it is easy to imagine a diagnostic process that includes refined tasks at later stages of the simulation and preliminary diagnostic decisions in the course of diagnostic activities. Consequently, the four components within the simulation overlap while maintaining their logical order. In the following sections, we describe the six components in more detail to establish the outer framework for our focal concept of the salience of the presented diagnostic information. Social Embeddedness Physical Environment Diagnostic Tasks Simulation Timeline Diagnostic Information •Access •Prominence •Simplicity •Clarity Diagnostic Activities Diagnostic Decisions Core Components Figure 1. Components of simulations of diagnostic situations https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 N. Machts et al., Categorization of simulated diagnostic situations 7 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) Physical environment The physical environment of simulations concerns the physical and technical conditions within the simulation itself and the external setup of the simulation. It involves the design approach (e. g., document analysis or interaction with simulated persons or groups), the simulation format (e. g., computer simulation, role-play, or video vignettes), and the external application of the simulation (e. g., in a classroom or laboratory setting). In summary, it refers to how information is presented to the participants and how they are enabled to interact with parts of the simulation. Social embeddedness The social embeddedness of simulations concerns the social, emotional, and motivational conditions both within the simulation itself and within its external setup. It involves relevant persons (e. g., employers, instructors, students, or collaborative partners), a reward structure (e. g., participation allowances or gamification elements), feedback mechanisms (e. g., feedback about correct diagnoses after completion of the assigned tasks), or personal relevance (e. g., participating in the simulation to acquire a certificate) and refers to why and with whom one should participate. It is important to stress that the technical component of social embeddedness concerns the conditions outlined in the design and application of the simulation. Actual measures of emotional and motivational states are dependent variables for these conditions. However, such measures can validate what the simulation design wants to achieve. Diagnostic tasks Diagnostic tasks concern the instruction participants receive in the initial or transitional phases of a simulation. These tasks can include the initial information from which diagnostic decisions follow and the prompting or recommendation of relevant diagnostic activities. Diagnostic tasks must initiate diagnostic activities; thus, we do not consider the direct request to make diagnostic decisions after acquiring diagnostic information to be a diagnostic task. Obviously, diagnostic tasks present an excellent opportunity for interventional experimental-controlled designs in simulations by giving groups of participants different instructions concerning their activities during the simulation (e. g., instructions may advise an experimental group to systematically collect information on multiple target characteristics, whereas a control group may receive no such instruction). Diagnostic activities Diagnostic activities are what the participants do during the simulation and include both actions as well as cognitive processes. In accordance with our minimalistic definition of diagnostic situations, diagnostic activities must entail identifying diagnostic information to support subsequent diagnostic decisions. In addition, many simulations require participants to generate diagnostic information through various available actions. Further diagnostic activities can include establishing diagnostic objectives for future actions, generating hypotheses that may influence further engagement with diagnostic tasks, and evaluating collected diagnostic information. Simulations can exercise control over a range of these cognitive activities (e. g., by modifying the presented information, or by prompting specific activities). Simulations can also explicate otherwise unobservable cognitive activities (e. g., by allowing or requiring participants to document their activities or their acquired diagnostic information), thus making these activities available as dependent variables. Considering that we excluded participants' emotional and psychological states as a component of the framework but included diagnostic activities, we may appear to be walking a thin line, as both are to some extent dependent variables. However, the range of possible actions associated with diagnostic activities is naturally constrained by the design of each simulation; this makes these activities a genuine family of controlled elements despite the possibility that actually performed diagnostic activities could be established as moderators and mediators of measures of diagnostic success. Diagnostic information Diagnostic information concerns stimuli that are perceivable by the participants during the simulation and that pertain in any way to a diagnostic target (e. g., a student's answer on a test or a patient's verbal description of a symptom). Diagnostic information is also an essential component of diagnostic situations, as it constitutes the foundation of the ensuing diagnostic decisions. Thus, it is the focal point of this framework, and the different facets of its salience are further elaborated upon in the section after the next. Naturally, complex stimuli often contain smaller pieces of information, which simulation designers can modify separately (e. g., features of long student essays or recorded conversations). To provide a comprehensive definition of information presented in a simulation, we differentiate between diagnostically relevant and irrelevant information: relevant pieces of information support accurate diagnostic decisions of interest in some way, while irrelevant information does not. Especially when a simulation requires multiple diagnostic decisions per target, pieces of information can be relevant to one decision and irrelevant to another. We limit this framework to the conceptualization of relevant information; however, it should also be extendable to irrelevant information, entailing a more https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 8 N. Machts et al., Categorization of simulated diagnostic situations Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) complex investigation of the interplay of relevant and irrelevant information that exceeds the scope of this paper. Diagnostic decisions Diagnostic decisions concern the participants' individual diagnostic products. They can include diagnoses (e. g., naming a medical illness or deciding on a student misconception), judgments (e. g., estimating a student's competence), predictions (e. g., predicting the course of an illness or estimating future performance), and follow-up actions (e. g., choosing a therapy or providing student counseling). Following our definition, a diagnostic situation must always include at least one explicit decision from either category; however, it does not necessarily include multiple explications (e. g., choosing explicit follow-up actions without preceding explicit judgments). If more than one explication is included, the logical order will be to arrive at an assessment and then to make a prediction or choose a follow-up action accordingly. In simulations, participants usually report diagnostic decisions in an open or closed questionnaire format; however, verbal reports or actual performances of follow-up actions are also possible in specific simulations (e. g., role play). In terms of measuring diagnostic competencies beyond knowledge testing, diagnostic decisions represent dependent variables and lead to the calculation of scores through comparison to success criteria. Salience of diagnostic information Salience is the property that makes stimuli stand out and be likely to draw the perceivers' selective attention (Higgins, 1996). Visual examples often include multiple stimuli, whereby salient stimuli stand out from others. Early research revealed that perceivers recall more information about salient stimuli (McArthur & Ginsberg, 1981) and that these stimuli receive disproportionate weighting in subsequent judgments (Taylor & Fiske, 1978). However, while researchers have found that people generally agree on which parts of a picture or scene are salient (Treisman & Gelade, 1980; Wolfe, 1997), individuals' attention can be directed through many different means. Thus, an important distinction is whether salience pertains only to the qualities of the stimulus or whether salience is a hypothetical construct of an individual's cognitive processes when perceiving stimuli. While Taylor and Fiske (1978) emphasized the importance of the individual point of view in their definition of salience, Higgins (1996) tried to differentiate between the generalizable attributes of stimuli and individual cognitive processes. In a simulation, the generalizable properties of stimuli, the guiding of cognitive processes through instructional design, and the participants' individual differences in perceiving the simulation come together. Consequently, we need a conceptualization of salience that allows for the generalizable categorization of the simulation design while maintaining the possibility of including participants' individual differences as moderators. To achieve this, we chose to establish a very broad concept of salience that includes multiple facets, targeting cognitive processing in diagnostic situations through the presentation of different kinds of information during a simulation. Our list of the four facets of salience includes the otherwise distinct concepts of access, prominence, simplicity, and clarity, where prominence represents salience in a narrower sense. Because we want to conceptualize how participants' attention is directed in simulations through the presentation of information, we coalesce these otherwise distinct terms to exhaust the targeted cognitive processes. These aspects are similar to Funder's (1995) realistic accuracy model, which proposes four facets of diagnostic information that lead to an accurate judgment: relevance, availability, detection, and utilization. Funder's aspects of availability and detection share features with our aspects of access and prominence. Howev er, we treat relevance as a precondition for considering a piece of information to be diagnostic in our framework, keeping it in mind for a possible extension that includes a model of the interplay of relevant and irrelevant information. Finally, we consider the utilization of diagnostic information as an outcome of its salience and, thus, a dependent variable in our approach. Access Similar to Funder's (1995) concept of availability, access determines whether relevant information becomes physically available to participants during a simulation. For a single piece of information, access is dichotomous: it is either overtly represented at one point in the simulation, or it remains hidden. Each diagnostic criterion – representing correct or accurate diagnostic decisions – can have a population of relevant information, but a simulation design may or may not provide access to this information. High access entails that participants are likely to identify the most relevant information during the simulation, whereas low access entails that a substantial amount of relevant information is likely to be withheld from or not retrieved by the participants. As we consider access in a physical sense and not in a cognitive sense, information can be physically accessible but likely to be missed. Cognitive accessibility more so factors into the other facets of salience. Prominence Prominence determines whether relevant information stands out and is likely to draw participants' attention durhttps://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 N. Machts et al., Categorization of simulated diagnostic situations 9 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) ing a simulation. In Funder's (1995) terminology, prominence is the driving factor behind information detection. It can be triggered by audiovisual markers of the pieces of information and their contextual presentation; temporal positioning or duration of presentation; and simulation plot sequencing. Additionally, diagnostic tasks, such as providing prompts on how to proceed with diagnostic information, can enhance its prominence. In contrast, the density of possibly irrelevant information can decrease prominence during a simulation. For a single piece of information, prominence is neither categorical nor definite. To determine the prominence of relevant information, we need to estimate it either theoretically (e. g., by asking simulation experts) or empirically (e. g., by asking the participants). Consequently, inter-individual differences concerning the prominence of pieces of information can be a threat to the reliability of its estimation. The lower the agreement on whether a piece of information in a simulation is prominent, the less reliable the point estimate of its prominence is. Nevertheless, individual differences in the perception of salience can lead to interesting research questions. One person may find a piece of information very alarming, whereas another person may barely notice it. Thus – in addition to its manipulation via simulation design – explicating participant's perceptions of the salience of stimuli makes prominence a dependent variable that can be used in mediation and moderation analyses. Overall, a high prominence of relevant information means that relevant pieces of information will stand out and participants will likely actively perceive it, whereas low prominence entails that relevant information will be presented in a way that makes it more likely for participants not to actively engage the information and possibly overlook or forget it. Simplicity Simplicity describes the amount and complexity of relevant information in combination with the number of diagnostic decisions. The amount and complexity result from the number of decodable pieces of information (e. g., a long student essay can be decoded into a very high number of single pieces, whereas a single test score is a very simple piece of information). The number of diagnostic decisions results from the number of targets (e. g., the number of students in a classroom or the number of patients that must be diagnosed in a round) and the number of decisions per target (e. g., the number of competencies to estimate per student or the required number of differential diagnoses per patient). High simplicity entails a few simple pieces of relevant information combined with few targets and few decisions per target. Low simplicity entails a heavy load of complex pieces of information that participants must attribute to multiple diagnostic decisions, possibly ranging across multiple targets. Clarity Clarity determines whether relevant information in the simulation objectively leads to precise and unambiguous diagnostic decisions. As such, it concerns the reliability of mapping relevant information in the simulation onto the respective decisions. In line with test theory, single pieces of information have relatively low reliability; however, this does not imply that a larger pool of information necessarily leads to unambiguous results. Information patterns may even seem to entail clear diagnostic decisions for participants but may be objectively ambiguous. For example, a diagnostic situation could include highly similar diagnostic information for two targets that require fundamentally different diagnostic decisions. Like prominence, clarity is neither categorical nor definite; thus, it must be estimated. This estimation could include using the agreement of experts on diagnostic allocations or decisions based on specific information. High clarity entails that relevant information will produce clear, unambiguous results when processed correctly, whereas low clarity entails the opposite. Example: The simulated classroom in Kaiser et al. (2013) In this section, we introduce one published example of a simulated diagnostic situation – Kaiser et al.'s (2013) simulated classroom – and categorize it using the introduced conceptual framework. In this study, the authors investigated the interplay of students' lesson performance and lesson engagement in participants' judgments in a simulated classroom environment for a sample of (n = 40) teacher candidates. A further description of the simulated diagnostic situation is embedded in the categories of our framework. Our focus is on exploring the salience of diagnostic information under the four separate headings below. It is important to note that within this focus, we apply heuristic wording categories to the four salience facets; we hope that this will provide an initial approach to deriving systematic scales in the future. The physical environment of this simulated classroom was an interactive computer application that led participants through a simulation using onscreen instructions. Static pictures and names represent simulated students. The participants interacted with these students using a question-answer schema. They clicked on questions in a given pool, and the students reacted by signaling that they want to provide an answer. Participants then clicked on any one student, and an answer appeared on the screen. After the simulated lesson was over, the participants made diagnostic decisions for each student by utilizing judgment scales. In terms of the external setup in the reported https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 10 N. Machts et al., Categorization of simulated diagnostic situations Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) sample, the researchers invited participants to a laboratory to perform the simulation on local computers in a standardized setting. The social embeddedness of this simulated classroom is explained in what follows. The participants interacted with the simulated students in a simplified manner. The simulated students were not personally introduced, and social information was gained only from pictures, names, and answers. There was no reward structure or feedback mechanism within the simulation. The external setup included standardized instruction from an experimenter, and the external reward structure involved payment for participation. Unsystematic feedback indicated that the participants felt the simulation was a moderately relevant representation of their later practice. The diagnostic tasks assigned at the outset of this simulation were to complete a lesson with a simulated class by selecting on-topic questions, addressing these questions to the class, calling upon students, and making judgments about the students at the end of the lesson. The tasks were not repeated or elaborated upon until diagnostic decisions were requested after the lesson. The diagnostic activities in this simulation included the identification and generation of diagnostic information, along with the available actions of choosing questions and selecting students to answer these questions. On a cognitive level, it was advantageous for participants to memorize students' answers and signaling frequency to make accurate judgments after the lesson; the active documentation of diagnostic information was prohibited. Participants were asked neither to generate additional diagnostic objectives nor to formulate hypotheses about the students during the simulation process. As students' answers were already marked in terms of whether they were correct or incorrect, the diagnostic activity of evaluating the information was not required. The salience of diagnostic information is at the core of our examination. Thus, we present details concerning the four facets of salience – access, prominence, simplicity, and clarity – in subsections following the next paragraph. Generally speaking, the relevant diagnostic information in this simulation was the students' signals after the participants directed questions to the class and the correctness of the students' answers when they were called upon. Additional irrelevant information likely arose from the students' names and pictures. The diagnostic decisions in this simulation involved judgments of the students' lesson performance and engagement. Using a percentage scale, the participants were asked to report for each student the relative frequency of correct responses in comparison to incorrect responses and the relative frequency of signals in comparison to the number of questions asked during the lesson, respectively. Access Access to relevant diagnostic information was high in this simulation, although this was due to the specific nature of the judgments. As described above, lesson performance was operationalized as the relative frequency of the correct answers for each student. Lesson engagement was operationalized as the relative frequency of the signals for each student. Participants called upon the students themselves and were shown whether the student's responses to each question were correct. They were also shown each student's signal. Thus, participants had access to every piece of information that constituted the diagnostic criteria, as the criteria were based on the diagnostic information the participants generated. Nevertheless, one can easily imagine how the simulated classroom could be used differently. For example, judgments could be requested regarding students' underlying competencies or abilities, while the limited number of calls on the students would restrict access to relevant information on the judgment criteria (i. e., fewer calls would entail less access to relevant information). Prominence As described above, the prominence of the relevant diagnostic information must be estimated. In this case, we estimate the prominence of diagnostic information in the simulation to be moderate. The students' signals and answers were central to the main diagnostic activity of generating diagnostic information without interfering with the information. Additionally, students' signals were visually highlighted on the computer screen using colored frames, and students' answers received additional colored highlighting based on whether they were correct or incorrect. However, the participants had to aggregate student answers into their post-lesson summative judgment without documentation and based only on memory. Thus, the prominence of each answer and its correctness likely diminished through the aggregation process until the end of the lesson. Simplicity The simplicity of the relevant diagnostic information in this simulation was moderate. Although the students' signals and the correctness of the students' answers had a simple and overt dichotomous nature, the participants directed various questions to the class during one lesson, resulting in a respective amount of answer information. Additionally, the participants received signaling information for each question and for each simulated student, resulting in a multitude of pieces of signaling information. This amount of information was combined with two distinct judgments made on each of nine students. Consequently, a high amount of very simple information had to be attribhttps://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66 N. Machts et al., Categorization of simulated diagnostic situations 11 © 2023 The Author(s) Distributed as a Hogrefe OpenMind article Zeitschrift für Pädagogische Psychologie (2024), 38 (1–2), 3–13 under the license CC BY 4.0 (https://creativecommons.org/licenses/by/4.0) uted to a moderately high number of overall 18 diagnostic decisions, which we consider to be moderately simple. Clarity The clarity of the relevant diagnostic information in this simulation was very high due to the nature of its design. The tasks used for the question-answer schema had solutions that were clearly correct or incorrect, and student answers were flagged accordingly in the simulation. This information was used for the aggregated diagnostic criterion of lesson performance. Furthermore, the students immediately signaled after a question was directed at the class. This signaling information was used for the aggregated diagnostic criterion of lesson engagement. Additionally, there was no source of objective unreliability or ambiguity in mapping the diagnostic information to the diagnostic criteria. If processed correctly, the diagnostic information would lead to an exact diagnostic decision. Discussion In the previous sections, we outlined our conceptual framework for categorizing simulated diagnostic situations, focusing on the salience of the diagnostic information in such simulations. We elaborated on six conceptual components – physical environment, social embeddedness, diagnostic tasks, diagnostic activities, diagnostic information, and diagnostic decisions – and proposed that there are four facets to the salience of the diagnostic information presented in simulations: access, prominence, simplicity, and clarity. We further used one in-depth example to demonstrate how to describe and categorize the design of a simulated diagnostic situation on a conceptual level and in terms of the salience of its diagnostic information. The application of simulations of diagnostic situations includes research on the structure and measurement of diagnostic competencies and on how their design as learning environments affects the advancement of such competencies. For both purposes, the existing studies and conceptual frameworks have merely touched upon simulation design principles concerning the presentation of diagnostic information (Chernikova et al., 2022; Heitzmann etal., 2019), while also reporting heterogeneity between the coefficients of diagnostic competencies and the effect sizes of simulations used as learning environments. Current meta-analytical research (Chernikova et al., 2024) indicates that the salience of diagnostic information can go beyond the existing findings related to other moderators in explaining the variance of learning gains in measures of diagnostic success. The presented conceptual framework enables a clear description and categorization of simulations of diagnostic situations and opens up possibilities to identify precisely how these simulations present diagnostic information to participants. The framework should provide opportunities for the systematic manipulation of the salience of relevant information in simulations and for targeting the diagnostic activities involved in dealing with more or less salient information through interventions. For example, interventions could aim (1) to guide participants to gain better access to relevant information, (2) to make important information more prominent, or (3) to retrieve diagnostic information that produces objectively clearer diagnostic results. A possible agenda for future research includes the following: a. the categorization of existing simulated diagnostic situations for conclusive summaries in a meta-analysis; b. the application of this framework to the design of new simulations to communicate on common ground concerning the application of simulations of diagnostic situations; c. the realization of systematic salience variations in existing and new simulations of diagnostic situations to investigate on an experimental level which aspects of information salience are responsible for differences in measures of diagnostic competencies; and d. the validation of interventions targeting information salience to derive recommendations for the facilitation of diagnostic competencies. Overall, we consider simulated diagnostic situations to be effective tools for research on the structure and measurement of diagnostic competencies and on the effects of the design of learning environments on the advancement of such competencies. However, the respective effects of simulations exhibit large variations that existing metaanalyses and frameworks cannot fully address. In this paper, we present an approach to drive the research on simulated diagnostic situations forward by enabling the systematic categorization of design principles, focusing on the salience of diagnostic information – a topic that has not yet received any systematic attention. With our framework, we offer other authors a reference for their simulation designs and suggest that failing to consider the salience of diagnostic information in specific studies or learning environments can lead to serious misinterpretations of the reported coefficients. References Allen, J., Buffardi, L., & Hays, R. (1991). The relationship of simulator fidelity to task and performance variables. https://econtent.hogrefe.com/doi/pdf/10.1024/1010-0652/a000364 - Monday, September 09, 2024 5:00:49 AM - Universitätsbibliothek Kiel IP Address:134.245.6.66