reDIRECT - Modernising Concept Inventories. A Case Study in Electrical Circuits

Practice Paper Recommended citation: Ridgway, L. M., Cox, T., Baneham, J., Westby, T., & Piga, F. (2025). reDIRECT - Modernising Concept Inventories. A Case Study in Electrical Circuits. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631828. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. REDIRECT – MODERNISING CONCEPT INVENTORIES. A CASE STUDY IN ELECTRICAL CIRCUITS L. M. Ridgway a, 1 , T. Cox b, J. Baneham c, T. Westby d, F. Piga e a Dublin City University, Dublin, Ireland, 0009-0006-9296-9231 b University of Nottingham, Nottingham, UK, 0000-0002-4070-6046 c Dublin City University, Dublin, Ireland, 0009-0002-5625-8396 d University of Nottingham, Nottingham, UK, 0000-0001-5640-175X e University of Nottingham, Nottingham, UK, 0009-0000-9461-2590 Conference Key Areas: Improving higher engineering education through researching engineering education Keywords: Concept inventories, Electrical Circuits, Student Understanding ABSTRACT Concept inventory questionnaires are tools which measure the conceptual understanding of students and can be used to measure the effectiveness of teaching interventions and innovations through a pre and post-test format. Using existing tools allows faculty to build upon existing research in the field, however sometimes it is necessary to adapt a questionnaire so it is suitable for the needs of students and educators. Many existing conceptual questions use filament bulbs which are now not commonly used in educational demonstration or electronics projects and so may be unfamiliar to modern student cohorts. A framework for updating concept inventories is presented with the update of an existing concept inventory (DIRECT) to “reDIRECT” used as a case study. reDIRECT was piloted with 601 participants. Test results were analysed using the statistical measures from the original paper which found reDIRECT has an average difficulty index = 0.37 (DIRECT = 0.41); average point biserial correlation = 0.34 (DIRECT = 0.32); average discrimination index = 0.29 (DIRECT = 0.23). Within the cohort differences were found in student understanding between those in the first year of a three-year BEng compared to those in the first year of a four-year BEng in all measures. The four-year BEng students study at universities in both Ireland and the UK, providing a reference point between the conceptual understanding of students at different entry points to degree programmes in engineering in each educational system. 1 Corresponding Author L. M. Ridgway [email protected] 1 INTRODUCTION The use of concept inventories gives a quantitative measure of the understanding of a cohort and the use of pre and post-tests can be used to evaluate the effectiveness of educational approaches in a range of disciplines. A large number of tools for this purpose have been developed over time, however these are not always suitable for direct deployment with a cohort. The purpose of this work is to provide a framework for the process of updating concept inventories through an example. While the case presented is from electronic engineering, the principles within the process are transferable to other disciplines. This allows the use of concept inventories that can assess the capabilities of modern students while remaining comparable with historical datasets. 2 CONTEXT AND PRACTICAL WORK 2.1 Concept and Concept Inventories The authors were seeking a tool which could measure learning gains to evaluate the effectiveness of their teaching methods and to establish the start of a data set to support future research in engineering education. All the institutions in this study have large classes (see Table 1 for cohort numbers), so a quantitative measurement instrument was the most practical to measure student understanding through a pre and post-intervention test. Student understanding of foundational engineering concepts is key to their progression (Fredette & Lochhead, 1980), (Halloun & Hestenes, 1985), (Shipstone et al., 1988) (McDermott & Shaffer, 1992), (Streveler et al., 2008) and a range of questions to probe student misconceptions were developed to explore student understanding of electrical concepts such as Cohen et al. (1983), Chang et al., (1998), Liégeois & and Mullet (2002). A wider range of tools have been developed for physics students which incorporate electrical concepts (Maloney et al., 2001), (Marx, 1998), (Allain & Beichner, 2004), however these also cover a wider range of areas which are less directly transferable to engineering courses. Three existing tests which focused on electrical circuits were identified as most relevant to the educational settings of the authors; the Electrical Circuits Conceptual Evaluation (ECCE) (Sokoloff, 1996), Determining and Interpreting Resistive Electric Circuits Concept Test (DIRECT) (Engelhardt & Beichner, 2004), and Inventory of Basic Conceptions – DC Circuits (IBCDC) (Halloun, 2007). The best fit for the curriculum within the participating institutions was DIRECT, which is a conceptual inventory test of direct current and circuit fundamentals. ECCE includes alternating current questions; and while IBCDC has similar material to DIRECT, there are no peer reviewed publications for this test. Studying the questions for DIRECT, it was felt that the tool could not be immediately deployed “off the shelf” as, while the underlying concepts remained relevant, it needed to be updated to include technical components that would be relevant and understandable for a student studying within a European university in 2024. The updated tool was named reDIRECT, with the work guided by the following research question: Can a concept inventory be updated while maintaining the reliability, correlation, discrimination and difficulty of the original? 2.2 A Design Process for Updating a Concept Inventory Using an existing concept inventory provides the advantage of the peer review process and prior results, meaning that it can be more impactful to adapt an existing diagnostic rather than designing a new one. Fig. 1 presents the process of the decision to develop reDIRECT, process steps are written inside of a box with two questions to emphasise the repeated reflection needed on these at all stages. Fig. 1. A design process for updating a concept inventory • Stage 1: A literature review or a repository such as PhysPort.org allows for scoping of existing tools. • Stage 2: If an existing inventory is not appropriate, then consider if questions can be updated to maintain the same goals as the original; revisiting the concepts of the original tool throughout the update process to ensure continued alignment between the old and new versions. • Stage 3: When the updated inventory is completed, use the instrument and complete statistical analysis to compare with existing data. 2.3 Updating Questions for reDIRECT Both DIRECT and reDIRECT contain 29 questions with five possible multiple choice answers, of which one is correct. Many of the questions in the diagnostic use screwthread lightbulbs. At the time of creation of the original DIRECT these components would have been in common usage in school science classrooms and as such, students sitting the test would have more familiarity with how these work where the brightness of the bulb is directly related to the current flowing through the filament, and where the screw thread is one of the two electrical terminals of the device. The ubiquity of light bulbs for science teaching in schools was uncertain in 2024. Certainly in university laboratories, light emitting diodes (LEDs) are the component of choice for most applications where a light is required, and it is rare to see filament bulbs in modern textbooks or other educational resources. However using LEDs has challenges for the purpose of reDIRECT as these are not directly linear devices (Eleftheriou et al., 2021). A visual representation for current in a circuit was important for the concepts in the questionnaire, so the authors chose to keep filament bulbs within the new version, but added an additional explanatory note on the front cover in case students were unfamiliar with the device. The original questionnaire includes two-dimensional sketches that represent filament bulbs with screw threads (Fig. 2) which are connected into circuits. Some of these questions were replaced with a new three-dimensional format lamp in holder format alongside a new three-dimensional battery (Fig. 5 includes an example). This generic form was chosen for clarity, as a bulb holder with a pair of distinct connection terminals is conceptually easier to comprehend without prior knowledge of the specific technology, and forms a generic two terminal circuit which is relevant for both historic and current (e.g. LED) bulb technologies. For circuit diagrams, the US style resistor symbols were replaced with IEC versions and the filament bulbs were replaced with generic lamp symbols. Fig. 2. Question 9 from DIRECT (Engelhardt & Beichner, 2004), which could not be meaningfully updated with lamps in holders so was replaced in its entirety with Fig. 3. Some questions in DIRECT look at the connections of a filament bulb when removed from its holder (questions 9, 13, 18, 22, 27). These cover a range of objectives which were revisited during question redesign to ensure the updated format remained aligned. Questions 13 and 22 concern the recognition of series and parallel connections. Question 27 tests understanding of short circuits. All of these questions were adapted to use three-dimensional component images. Questions 9 (Fig. 2) and 18 (Fig. 4) evaluate student understanding of the twoterminal nature of circuit elements. Question 18 was updated with new illustrations (Fig. 5), however question 9 could not be meaningfully updated due to the physical connection between the battery and filament bulb in the original. Because LEDs are more visibly two-terminal devices and because the physical connections of traditional screw bulbs are less relevant for students to understand in the 21st century, this question was replaced to ask students the same question as in question 18, but with the connections represented by circuit schematics not illustrative components (Fig. 3). This also allows comparison of student understanding between different representative forms of circuit connections. Fig. 3. Question 9 for reDIRECT. This replacement for question 9 in DIRECT (Fig. 2) is not the same question and instead is a schematic representation of the circuit configurations in question 18 (Fig. 5). Fig. 4. Question 18 for DIRECT (Engelhardt & Beichner, 2004) with the original battery, bulb and wire illustrations. Fig. 5. Question 18 for reDIRECT with the new battery, lamp and wire illustrations. 2.4 Pilot Study Cohorts reDIRECT was conducted with three student cohorts during the first semester of the 2024/25 academic year. All participants are in their first year of university but studying on different programmes in different contexts (Table 1). The suffixes -1 and -2 indicate the first and second rounds of running the questionnaire with the same class group. The first run was conducted at the start of the academic year to attempt to capture the understanding of students upon entering university. The second round was conducted after some fundamental circuits content had been covered within modules, but still during the semester to balance the need for a sufficient sample size. Participants had 30 minutes to complete the questionnaire. For CE-2 the questions were viewed online with students completing a machine readable paper based multiple choice response sheet. The EEE and FEPS cohorts were given a paper based question sheet where students could indicate their answer on the same paper, which was then manually marked. Table 1 - Cohorts Participating in reDIRECT. Cohort Code University Programme of Study Number of participants EEE-1 University of Nottingham Electrical & Electronic Engineering Year 1 of 3 year BEng 144 EEE-2 University of Nottingham Electrical & Electronic Engineering Year 1 of 3 year BEng 56 FEPS-1 University of Nottingham Foundation Engineering & Physical Sciences Year 1 of 4 year BEng 205 FEPS-2 University of Nottingham Foundation Engineering & Physical Sciences Year 1 of 4 year BEng 105 CE-2 Dublin City University Common Entry into Engineering Year 1 of 4 year BEng 91 3 RESULTS AND INSIGHTS 3.1 Summary Statistics The questionnaire data was considered by cohort and then combined as a whole. Statistical analysis was performed using the same measures as in DIRECT version 1.1 tested on American university students (Engelhardt & Beichner, 2004) to allow direct comparison between the tests and presented in Table 2. Table 2 - Statistical analysis for reDIRECT cohorts compared with DIRECT reDIRECT Cohort DIRECT v1.1 Target value EEE-1 EEE-2 FEPS-1 FEPS-2 CE-2 All Cohorts Sample size 144 56 205 105 91 601 692 Mean 42% 55% 32% 34% 34% 37% 41% Range 0 – 83% 10 – 86% 10 – 72% 0 – 79% 10 – 69% 0 – 86% 3.4 – 90% 0 – 100% KR-20 0.73 0.84 0.51 0.69 0.65 0.74 0.70 0.7 – 0.9 Average pointbiserial correlation 0.35 0.42 0.25 0.30 0.28 0.34 0.32 ≥ 0.20 Average discrimination index 0.25 0.29 0.22 0.28 0.21 0.29 0.23 ≥ 0.30 Average difficulty index 0.42 0.55 0.32 0.34 0.34 0.37 0.41 0.4 – 0.6 The overall mean for reDIRECT (37%) is lower than DIRECT (41%). However the variation between the cohorts provides a more detailed picture where the already specialised EEE-1 and EEE-2 cohorts have significantly higher means. The high school mean for DIRECT (36%) is more comparable with the FEPS and CE cohorts as these groups are earlier in their academic careers and composed of a large proportion of non-specialists who will not continue to electrical and electronics engineering or physics courses. This aligns with the prior findings using DIRECT that the intended major of students is significant in test outcomes (Sangam & Jesiek, 2012). The distribution of scores for reDIRECT as a whole is provided in Fig. 6. The reliability of the questionnaire was evaluated using Kuder-Richardson Formula 20 (KR-20). This measure considers the consistency of all the questions on the test to each other. Point bi-serial correlation indicates the degree of correlation between a student answering a particular question correctly with their overall test score. An item with a low value indicates that answering a single question correctly is not correlated with getting a higher overall score. The discrimination index reports the ability of an individual question to discriminate between participants who fall within the top 27% or bottom 27% on the test overall. The difficulty index is the percentage of participants who answer an individual question correctly (therefore the average difficulty index is the same as the mean). Fig. 6. Distribution of scores for all participants in reDIRECT. For reliability, correlation, discrimination and difficulty, the EEE cohorts scored closer to the reported results of DIRECT and to the preferred statistical ranges. This implies that the tool may be less appropriate for usage in general engineering cohorts; the results in Table 2 show in a variety of measures they find the questions more challenging as a group. However the results are within the acceptable bounds so the questionnaire still provides value, as it does provide quantitative evidence that the two general entry programmes have comparable cohorts across the two countries. 3.2 Question Analysis Table 3 presents the analysis for each question across the old and new instruments. Table 3 - Question analysis for reDIRECT and DIRECT. Questions 9 and 18 underwent significant changes and are highlighted with black. Other questions where 2D filament bulb sketches were replaced by 3D illustrations are highlighted with grey. Question reDIRECT DIRECT Correlation Discriminati on Difficulty Correlation Discriminati on Difficulty 1 0.26 0.21 0.34 0.28 0.23 0.38 2 0.27 0.41 0.05 0.25 0.07 0.07 3 0.30 0.23 0.39 0.38 0.32 0.46 4 0.40 0.40 0.28 0.35 0.32 0.37 5 0.42 0.38 0.36 0.44 0.38 0.39 6 0.39 0.27 0.53 0.33 0.29 0.54 7 0.36 0.29 0.45 0.41 0.35 0.51 8 0.34 0.14 0.77 0.35 0.25 0.74 9 0.48 0.51 0.24 0.44 0.35 0.72 10 0.31 0.22 0.37 0.25 0.17 0.34 11 0.00 0.00 0.12 0.00 0.01 0.04 12 0.33 0.44 0.18 0.41 0.21 0.20 13 0.38 0.17 0.73 0.33 0.20 0.82 14 0.47 0.36 0.41 0.52 0.43 0.41 15 0.44 0.31 0.49 0.31 0.22 0.49 16 0.25 0.19 0.53 0.17 0.14 0.57 17 0.52 0.46 0.34 0.41 0.32 0.43 18 0.43 0.42 0.25 0.29 0.18 0.46 19 0.37 0.22 0.60 0.38 0.29 0.62 20 0.08 0.11 0.13 0.10 0.03 0.14 21 0.33 0.19 0.56 0.27 0.19 0.52 22 0.40 0.22 0.65 0.33 0.27 0.44 23 0.37 0.37 0.29 0.36 0.26 0.40 24 0.49 0.57 0.22 0.43 0.29 0.24 25 0.23 0.37 0.05 0.20 0.05 0.05 26 0.50 0.44 0.37 0.42 0.32 0.40 27 0.39 0.16 0.77 0.39 0.30 0.73 28 0.11 0.17 0.23 0.13 0.06 0.24 29 0.27 0.31 0.18 0.22 0.16 0.19 Average: 0.34 0.29 0.37 0.32 0.23 0.41 Question 18 reports results that the difficulty has decreased in the new format, while the correlation and discrimination have increased. The new format makes the twoterminal nature of a lamp more visible, so this is not unexpected. Looking at question 9, the two studies cannot be directly compared as the question has been replaced with a lower difficultly question, however the correlation and discrimination index are within the acceptable range. Comparing the reDIRECT results for question 9 and question 18 which are the same question with different circuit representations, of note is the higher discrimination index for the version asked with a circuit schematic, indicating that higher scoring students are more likely to answer correctly when a question involves interpreting a circuit diagram over a visual stand-in. Updates to question 13 show only small variations in the measures. Question 22 appears to be more difficult to participants than the previous version, whereas question 27 has a lower discrimination index than previously. These differences were unexpected and will be the subject of follow-up research both through analysing further runs of reDIRECT and by interview work with students. 4 CONCLUSIONS AND IMPLICATIONS Addressing the original research question, this work presents evidence that it is possible to update an existing concept inventory while maintaining the core value of the tool (as demonstrated by statistical analysis). The design process is presented to support others in adapting concept inventories from their own disciplinary areas. reDIRECT is the successful modernisation of a tool which can be used to assess knowledge and understanding in electrical circuits across multiple cohorts and educational models. It provides educators with information on what misconceptions a class holds and provides a pre and post-test structure to measure the success of curriculum interventions. A planned future expansion of this work is to repeat the original DIRECT with one of the cohorts to better understand the effect of updated symbols and use qualitative methods to gain insights into how students comprehend circuit components.