Novel Adaptive Entry-Level Maths Assessment Test for Sorting Students to Different Level Courses

Research Paper Recommended citation: Kun, L. Á., Berezvai, S., Lógó, E., & Szilágyi, B. (2025). Novel Adaptive Entry-Level Maths Assessment Test for Sorting Students to Different Level Courses. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631398. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. NOVEL ADAPTIVE ENTRY-LEVEL MATHS ASSESSMENT TEST FOR SORTING STUDENTS TO DIFFERENT LEVEL COURSES László Ákos Kuna, Szabolcs Berezvai b, 1 , Emma Lógó c, Brigitta Szilágyi d a Budapest University of Technology and Economics, Budapest, Hungary b Budapest University of Technology and Economics, Budapest, Hungary 0000-0002-6399-583X c Budapest University of Technology and Economics, Budapest, Hungary d Budapest University of Technology and Economics, Budapest, Hungary; Corvinus University of Budapest, Hungary; MTA–ELTE Theory of Learning Mathematics Research Group, 0000-0002-2566-0465 Conference Key Areas: Teaching mathematics and physics in engineering education, Keywords: Mathematics in engineering, Assessment test, Competence in mathematics, Adaptive Testing, Dropout ABSTRACT In engineering programs significant differences can be observed in the mathematical proficiency of students at the beginning of their university studies due to the considerable disparity between the content of the intermediate and advanced level math classes in the Hungarian high school system. To place incoming students in courses that match their knowledge in large-scale courses where the number of students exceeds 100, an effective assessment test is needed that not only evaluates the procedural knowledge but also the competences in mathematics. In the fall of 2024, a comprehensive entry-level assessment was conducted among incoming freshmen, involving a total of 2,196 students attending to six different faculties. This contribution aims to present a novel adaptive entry-level math assessment test for freshmen, which was performed before the start of the first-year Calculus courses. The test consisted of three parts, where the results of each part were used to select the set of questions to be asked in the next section. By analyzing the results of the three-part adaptive test, we aim to explore how incoming students can be grouped based on their knowledge levels. This categorization enables us to recommend either the A or B level mathematics courses, best suited to their proficiency. 1 INTRODUCTION Years of experience have shown that incoming freshmen arrive at universities with very different levels of maths knowledge. One reason for this is that the Hungarian admission requirements allow entry through various types of prior studies, so even in high school, students’ educational paths could differ significantly, with each student 1 Corresponding Author Szabolcs Berezvai [email protected] having taken advanced courses in different subjects. As a result, the math’s knowledge level spans a very wide spectrum, with an enormous gap between the weakest and the strongest. This requires different course materials and teaching methods in engineering higher education. Since these differences are often not reflected in the admission scores, it becomes necessary to assess the incoming students’ knowledge levels to implement differentiated education. Another reason for the weaker entrance performance among first-year students can be attributed to the long-term effects of the pandemic. The COVID-19 pandemic posed an unprecedented challenge even for university instructors in modern education. Research from previous years showed that the difficulties did not disappear with the return to in-person teaching (Sipos et al., 2019, Berezvai et al., 2024). It was expected that even in 2024, the traces of the pandemic would still be evident in the performance of incoming university students. To address this issue, the BME Department of Mathematics decided to implement differentiated math education. In September 2024, teaching Calculus began on two levels. The “A” level is recommended for students who already possess the mathematical knowledge required for engineering studies, while the “B” level is for those who have significant deficiencies. The latter group may lack the necessary knowledge either because they did not attend the advanced courses or because they did not fully acquire the necessary concepts there. At the “B” level, in addition to the first-semester calculus material, efforts are made to fill in the gaps in knowledge. The goal is not to segregate students but to provide a supportive learning environment that fosters the development of students with varying capabilities. Differentiated grouping can help ensure that every student begins their university studies at the level most appropriate for them and can gradually catch up to the higher-level requirements. The effectiveness of differentiated instruction is supported by numerous studies. The comprehensive analysis by Smale-Jacobse et al. (Smale-Jacobse, 2019) demonstrated that differentiated education in high school has a positive impact on students’ academic performance, especially when teaching methods are adapted to individual learning needs. Their research suggests that differentiated instruction not only results in more stable and deeper knowledge but also produces better long-term academic outcomes. Furthermore, the study of Young indicates that a differentiated approach can facilitate a seamless transition between high school and higher education by allowing for the integrated development of students with diverse levels of preparedness (Young, 2006). 1.1 Motivation and goals The purpose of the entrance assessment was to reveal the level of high school mathematical knowledge among students entering the university, since the significant differences in the levels of knowledge acquired in high school education, which pose direct challenges for universities. In response, the university is adapting by forming differentiated educational groups based on students’ knowledge levels, providing opportunities for students with varying backgrounds to develop their mathematical skills at their own pace and level. The data collected during the assessment enables a systematic analysis of the knowledge students acquired during their high school studies. To place incoming students in courses that match their knowledge in large-scale courses an effective assessment test is needed that not only evaluates the procedural knowledge but also the competences in mathematics. As a solution a novel online adaptive entry-level math assessment test was introduced. The test consisted of three parts, where the results of each part were used to select the set of questions to be asked in the next section. By analyzing the results of the three-part adaptive test, we aim to explore how incoming students can be grouped based on their knowledge levels. This categorization enables us to recommend either the A or B level mathematics courses, best suited to their proficiency. 2 METHODOLOGY 2.1 Test system: EduBase The entrance testing was carried out using EduBase (www.edubase.net), an online platform that offers customizable testing and learning solutions (Szilagyi et al., 2020). One of its outstanding features is the real-time data collection and feedback system, which allows instructors immediate access to information related to students’ performance. EduBase also utilizes advanced anti-cheating technologies to ensure the reliability and credibility of the tests. Additionally, the platform provides parameterizable questions, making it easy to customize and adapt exam questions. EduBase stands out for its full scalability, allowing it to be easily used by groups ranging from small classes to large corporations. The system integrates seamlessly with various educational systems, connecting with existing platforms such as LMSs or other corporate systems. This ease of integration and flexible management enables EduBase to perform effectively in different educational settings, from school education to corporate training (Berezvai et al., 2019). 2.2 Adaptive assessment test The adaptive test is an assessment method designed to evaluate students’ knowledge and abilities, particularly in mathematical competencies. The essence of the test is that the difficulty level of the questions dynamically adjusts based on the responses, so that throughout the examination, the questions that best match the student’s actual knowledge and skills are presented. The main features of the assessment tests are • Adaptive Questions: During the test, the difficulty of the question set varies according to the student’s responses. • More Accurate Assessment: students are measured based on questions that best suit their capabilities ensuring more accurate evaluation. • Motivating Effect: Adaptive testing is generally more motivating for students, as it avoids tests that are too hard or too easy. • Time Efficiency: Due to its adaptivity, the test can usually be completed in a shorter time compared to traditional, static tests. During the test, students were allowed to use only paper and pencil without calculator, which simplified the administration and review process. 2.3 Adaptive assessment test structure In the first phase (Basic Questions), the students’ general mathematical knowledge and procedural calculation skills was measured with questions based on basic operations, identities, linear equations, inequalities, basic trigonometric functions and elementary analytic geometry. Most of the tasks were solvable with 10th grade knowledge. In this part, a total of 20 questions had to be answered within 10 minutes. Subsequently, the students were divided into two groups based on whether they had attended optional mathematics classes (i.e. Facultation) during their high school studies or not (Facultation (F) or Non-Facultation (NF)). In this phase, the students had to solve 15 or 16 questions, and they were given 50 minutes. The questions fully covered the high school mathematics curriculum, and some questions were identical for both groups to obtain a more accurate picture of the students’ knowledge level, as well as to determine the borderline between the two groups. After the tests were completed, the EduBase system provided immediate feedback, which allowed us to immediately assign the students to the next level. In the third phase, the students were divided into four groups, which were named as follows: Facultation+ (F+), Facultation- (F-), Non Facultation+ (NF+), and Non Facultation- (NF-). The classification was based on whether the test taker achieved below or above 50% in the second block. If the performance was above 50% and the student had attended advanced classes, they continued the test at the F+ level; if below, then at the Flevel. For students who did not attend advanced classes, the NF+ and NFlevels were similarly established. In this phase, the number of tasks varied, although an identical 40-minute time frame was provided for every group. This multi-level assessment enabled a more precise measurement of the students’ high school knowledge and the formation of differentiated groups that accurately reflect the students’ prior mathematical knowledge and the differences arising from their high school studies. In this phase, similarly to the previous level, students received some questions from the test set of the adjacent higher or lower-level group. This method allowed for a more precise delineation of the blurred boundaries between the groups, providing a more detailed picture of the finer differences in students’ knowledge levels. Fig. 1. The testing process with the number of questions and the allocated time A sample test of all test levels is presented in the Appendix that can be downloaded from the following link: https://bit.ly/448wY8x 3 RESULTS In the fall of 2024, a comprehensive entry-level assessment was conducted among incoming freshmen. Six faculties with 12 engineering programs (civil, transportation, vehicle, chemical, biotechnology, environmental, electrical, mechanical, energy, mechatronics, industrial design and computer engineering), 2 natural sciences programs (mathematics, physics), involving a total of 2,196 students. Our goal was not only to assess students’ knowledge but also to create clusters so that students could be placed in a group corresponding to their proficiency level from the first week of the academic year. To achieve this, we had to establish a ranking method that would not collectively distort any group but would clearly determine the appropriate placement for students near classification boundaries. For example, if a student who did not attend advanced mathematics classes performed well at the second and third levels, then despite not having taken an advanced mathematics course, if they answered correctly the common questions from the higher-level set, we needed to reward this to some extent. To develop this model, we first needed to examine the raw results. 3.1 Procedural Skill Assessment results – Basic Questions This was the first part of the entrance assessment, completed by 1372 students. The average score was 58.72%, with a standard deviation of 19.4%. Fig. 2/a illustrates the distribution of results, which shows that the results follow a normal distribution. This section was successfully completed by students with strong computational skills. Given that the time limit was 10 minutes, approximately 30 seconds were available per question. Incorrect answers often resulted from common conceptual errors, which allowed us to analyze typical mistakes after evaluating the test. Fig. 2. Results Block 1: a) histogram of results and b) change of results by tasks E.g. the sixth question of the test was as follows: What is the value of 𝑙𝑜𝑔3(27)? Students are expected to be familiar with logarithms at an intermediate level. In this case, the argument of the base-3 logarithm was a commonly used power of three. Therefore, those who understood the concept could recognize the result immediately. Fig. 2/b presents the percentage distribution of responses for the first 20 questions, categorized into correct answers, incorrect answers, unanswered questions, and unopened questions. The results indicate a gradual decline in the proportion of correct answers from the first question onward. This is natural, as the difficulty of the questions increased over time. While the first question had a correct response rate of 94.04%, by the end of the test, this rate had dropped to only 11.41%. This steady decline suggests that student performance deteriorated as the test progressed. The rate of unanswered questions also followed an increasing trend. As the test progressed, more students left questions unanswered, especially toward the end. This suggests that time constraints may have played a role in students’ ability to complete the test. The proportion of unopened questions showed a consistent increase. This trend implies that, students increasingly skipped questions without even opening them. This result aligns with the declining proportion of correct answers and the rising number of unanswered questions, suggesting that students’ focus, or motivation diminished over time, and time management may have been a significant challenge. Overall, the trends observed in the Fig. 2/b suggest that student performance declined as the test progressed, supported by the decrease in correct responses and the increase in unanswered and unopened questions 3.2 Second block (NF and F) The second block contains exercises that students may have encountered during their secondary school studies. There were no unusual tasks in this part, they had to apply what they had learned. Fig. 3. Results Block 2: a) NF and b) F groups The number of tested students in the NF group was 402. The average result of this group was 35.93 ± 16.84%. In contrast, 969 students were tested in the F group, where the following result was obtained: 51.4 ± 13.34%. It is also evident that nearly onethird (29.3%) of first-year students did not attend elective courses, meaning that a significant number of students lack essential knowledge (e.g., trigonometric functions), which is crucial for successful performance in higher technical education. Looking at the histograms (see Fig. 3.), it is also apparent that students who did not attend Facultation performed poorly, with slightly more than a third achieving results between 30-40%. The number of students scoring above 70% was minimal. Nearly three-quarters of them did not reach 50%, meaning they continued the test on the NFbranch. 3.3 Third block (NF-, NF+, F-, F+) The results of the four different tests (i.e. NF-, NF+, Fand F+) are shown in Fig. 4, while the results are listed in Table 1. It is evident that of the 287 students progressing on the NFbranch, approximately 40% again scored between 30-40%, and there were no performances above 80%. Most students on the NFbranch struggle with significant knowledge gaps, making the reinforcement of high school knowledge a crucial element in their calculus education. The average score of students tested on the NF+ branch was not much higher than that of those on the NFbranch, but they received more difficult tasks. Their results were not particularly good either, as only a few achieved scores between 70-90%, and there were no performances above 90%. The 37.4% of the students attending the specialization course scored above 50% and were placed in the F+ group. It can be said that both groups performed better in the specialization track than in the non-specialization track. The specialization tests included parts of the material (e.g., differential calculus) that the students will encounter in their first semester at the university. For them, much of the material from single-variable calculus will be familiar. The Ftest had more procedural tasks (e.g., calculating the value of the derivative at a given point), while among the F+ tasks, there were questions that were unusual for Hungarian students, testing a deeper understanding of the concepts. Fig. 3. Results Block 3: a) NF-, b) NF+, c) Fand d) F+ groups 4 CLUSTERING STUDENTS During the sorting of students to A or B levels group limits also had to be considered, so it was necessary to define a ranking that would make the results of testers at different levels comparable. For this the various test results needed to be weighted in such a way that individuals with good abilities, but modest prior knowledge would not be placed in a lower level. Table 2 provides information about the applied weights. Table 1. Weights of various test applied for clustering students Block 1 Block 2 Block3 Test Weight Test Weight Test Weight Basic 1.0 NF 0.9 NF0.9 NF+ 1.0 F 1.1 F1.1 F+ 1.2 Considering the performances in different parts with various weights, a single score could be assigned to each student. The final score of the assessment test is shown in Figure 4. The histograms show that students who took the test designed for specialization courses generally performed better, resulting in a distribution that shifted to the right. In contrast, in the groups where students did not attend Facultation classes, the distribution shifted to the left, indicating lower average performance. It is worth highlighting that in the Non-Facultation plus (NF+) group, the test result distribution was nearly identical to the results of the two specialization groups. This suggests that although these students did not receive advanced high school mathematics education, they achieved similar results to their specialization counterparts on the entry test. 4.1 Applied clusters For sorting the students, a clustering method involving the admission points and the adaptive assessment tests results was applied. For the admission points (total of 500 points), we determined two key threshold values: 350 and 400 points, while for the entry test results, the 40% and 70% values were used. The results shows that the F+ group has high admission points and strong test results. A similar trend can be observed for the NF+ and Fgroups as well. An especially interesting observation is that in the NFgroup, high admission points are found despite none of the students achieving the 70% threshold on the entry test, and in fact, only a few came close to the 50% performance level. This leads to the conclusion that admission points alone do not provide enough foundation to assess the expected success during university studies, and they also offer too little guidance for implementing differentiated education. The groups were formed so that level A included students with stable knowledge sufficient to master the university curriculum, while level B included students who also needed time to cover the high school curriculum during the course. Fig. 4. Student clusters based on their admission points and assessment test results 5 CONCLUSION Given that mathematics plays a crucial role in the success of engineering studies, it is essential that students begin their studies with a solid foundation. Therefore, it is necessary for students to receive differentiated, remedial mathematics education tailored to their knowledge level in the first semester to provide a solid foundation in Calculus during the first semester after filling in any missing knowledge. By the end of the semester, every student must be well-prepared not only for the multivariable calculus and linear algebra courses in the spring semester but also equipped with mathematical knowledge that allows them to apply what they have learned in mathematics to their major subjects. Our adaptive measurement indicates significant