SETEASY— A Multi-Modal Classroom Engagement Assessment and Seating Optimization Framework

SETEASY A Multi-Modal Classroom Engagement Assessment and Seating Optimization Framework ZHIHAO XIE, HONGYE YANG Wuxi Taihu University [email protected] Beijing Institute of Architectural Design Co., Ltd [email protected] AND SHIEN LIU Wuxi Taihu University [email protected] Abstract. SetEasy optimizes classroom engagement in fixed seating grids. It fuses multimodal sensing (wristband physiology, 4K video, environmental data) and trains a v-Gage model grounded in a revised ISEQ. Each week, two-week engagement forecasts are mapped to a student–seat utility matrix, and CP-SAT generates seating plans under visual-access and social-dynamics constraints. In a four-week deployment (23 students, 331 classes), v-Gage converged across affective, behavioral, cognitive, and overall dimensions, cutting RMSE from 0.75 to 0.53. Optimization raised mean engagement from 0.30 to 0.70, with over two-thirds of seats reaching high engagement and backrow low-activity patterns markedly reduced. These results show that, without hardware changes, interpretable, data-driven seating strategies can substantially enhance engagement. The multimodal “assessment + optimization” paradigm offers a transferable, sustainable path to culturally responsive, differentiated spatial design amid global homogenization. Keywords: Machine Learning, Educational Space Optimization, Computational Design, Multimodal Sensing, CP-SAT Optimization 2 ZHIHAO XIE, HONGYE YANG AND SHIEN LIU 1. Introduction Under the wave of global educational standardization, schools worldwide have largely retained the “grid-based” seating arrangements originating from Western educational systems. This practice frequently results in mismatches between classroom furniture dimensions and students’ body types or cultural needs(Castellucci, Arezes and Viviani, 2010). In recent years, primary and secondary education reforms have called for a shift in learning environments from unidirectional instruction to learnercentered models that emphasize individual differences and foster active collaboration(Barrett et al., 2015). Extensive research indicates that classroom seating arrangements and spatial design have a significant impact on students’ learning outcomes and classroom engagement(Mäkelä and Leinonen, 2021). For example, Barrett et al. found that simply optimizing seating layout can yield up to a 16% improvement in learning efficiency. However, limited by school infrastructure and funding, most schools still rely on fixed desk layouts, making it a practical challenge to optimally assign “who sits where” within the existing grid(Peponis et al., 2007). Currently, seat assignments in classrooms primarily rely on teachers’ experience—either by making manual adjustments based on guidelines and shortest-path methods(Meller and Gau, 1996), or by iteratively experimenting with adjacency matrices in CAD software(Liggett, 2000). When balancing multiple objectives such as capacity, visibility, and social dynamics, this process is time-consuming and rarely achieves a globally optimal outcome. Fixed seating layouts have also been shown to inhibit collaboration and behavioral engagement(Kariippanon et al., 2019). In response to these limitations, research in computationally intelligent seat assignment has advanced rapidly. Genetic algorithms are capable of searching thousands of combinations at once, significantly improving satisfaction(ShinIke and Iima, 2016); Merrell et al. mapped interior design principles into probability density functions to interactively recommend seating positions(Peponis et al., 2007); deep reinforcement learning has modeled seat swaps as a Markov process, converging on multi-objective optima(Kakooee and Dillenburger, 2024); and the EDU-AI framework applies attention heatmaps in closed-loop seating optimization(Kariippanon et al., 2019). However, most existing methods depend on single behavioral factors or static preferences and lack real-time integration of multimodal data—physiological, behavioral, and environmental. Meanwhile, educational scholars have long criticized “one-size-fits-all” classroom models for undermining students’ cultural identity and learning motivation(Ladson-Billings, 2014). Culturally responsive teaching stresses a deep understanding of learners’ sociocultural backgrounds(Gay, 2002), and SETEASY 3 self-determination theory emphasizes that environments must satisfy students’ needs for autonomy, competence, and relatedness(Pekrun and LinnenbrinkGarcia, 2012). The OECD likewise advocates for locally data-driven design to break the homogeneity of seating arrangements(OECD, 2013). Yet, teachers working in classrooms with fixed desks and chairs still lack quantitative tools for implementing differentiated seat assignment. To bridge this gap, we present SetEasy—an intelligent framework for classroom seating optimization. The system first integrates physiological and behavioral data from students and teachers, as well as environmental information, and incorporates questionnaire results to construct an engagement prediction model. Combined with seat calibration data, these inputs are used to generate a student-seat utility matrix, enabling the generation of optimal seating plans through a weekly closed-loop iteration. In real-world deployment, after seven cycles, overall classroom engagement increased by approximately 45%, validating that data-driven seat assignment can significantly enhance educational outcomes in traditional fixed seating environments. Furthermore, SetEasy provides a scalable and replicable technical pathway for implementing localized and culturally responsive classroom practices. 2. Methodology SetEasy is an intelligent, multimodal perception-based framework for optimizing seating arrangements within fixed classroom grids. The system first collects data from wristband physiological sensors, 4K behavioral video, and environmental monitors, and integrates these with ISEQ questionnaire responses to build the v-Gage engagement prediction model. Engagement predictions from the previous two weeks are fused with seat calibration data to generate a student-seat utility matrix. Finally, CP-SAT integer programming is used to maximize utility under teacher-defined constraints, 4 ZHIHAO XIE, HONGYE YANG AND SHIEN LIU such as vision and social dynamics, outputting the optimal seating plan in a weekly closed-loop iteration (see Figure 1). Figure 1. Classroom Engagement & Seating Flow. 2.1 DATA COLLECTION Data were collected in June 2025 from a standard classroom at a secondary school in a medium-sized city in southern China. Over a four-week period, 23 first-year high school students (13 female, 10 male) and 6 teachers participated as study subjects, resulting in 331 valid classroom session datasets. 2.1.1 Physiological and Behavioral Data Collection Three types of sensors were deployed for data collection (see Figure 2). First, each student wore an Empatica E4 wristband to continuously capture multichannel physiological signals during class, including electrodermal activity (EDA) at 4 Hz, blood volume pulse (BVP) at 64 Hz (used to derive heart rate variability, HRV, metrics), 3-axis acceleration (ACC) at 32 Hz, and peripheral skin temperature (ST) at 4 Hz(Gao et al., 2020). Second, to obtain fine-grained behavioral data, the system followed the approaches of the StuArt model(Zhou et al., 2023) and the open-source Student Classroom Behavior dataset(Yang, Wang and Wang, 2023), performing real-time recognition of five key classroom behaviors—handraising, standing, writing/reading, dozing, and mobile phone use —at a frequency of once per second. A 4K wide-angle camera was mounted at the top front of each classroom to continuously record and analyze classroom dynamics at one frame per second. SETEASY 5 Figure 2. Diagram of Physical Deployment in Classroom. Additionally, each classroom was equipped with a Netatmo environmental sensor, automatically recording indoor CO₂ concentration, temperature, humidity, and noise level every five minutes to provide real-time assessment of environmental impacts on students. The overall data sensing timeline is shown in Figure 3. We implement multiple privacy safeguards: (1) guardians and students provide written informed consent; (2) video and wearable data are processed exclusively via on-premises, offline inference on school computers—raw data are never written to disk and are immediately deleted, with only de-identified weekly features and seat scores retained; (3) students participate under randomly assigned ten-digit IDs, reducing re-identification risk.2.2 2.1.2 Engagement Annotation and Ground Truth Construction In this study, “classroom engagement” is defined as a composite of students’ focus and willingness to participate in classroom activities across three dimensions: (1) Cognitive engagement: the depth of students’ understanding and critical thinking about classroom content; (2) Affective engagement: the level of interest and enjoyment students experience during lessons; 6 ZHIHAO XIE, HONGYE YANG AND SHIEN LIU (3) Behavioral engagement: students’ attentiveness and active participation in class. Although multimodal sensor data provide objective measurements of students’ behaviors and physiological states, subjective self-report measures remain crucial for supervising the predictive model. To ensure reliable ground truth, we adopted the validated In-Class Student Engagement Questionnaire (ISEQ)(Fuller et al., 2018). Consistent with previous research using wearables in classroom settings, which adapts survey items for different age groups(Di Lascio et al., 2017), we simplified the original ISEQ—designed for college students—to better suit the cognitive level of high school students. Additionally, based on recommendations from Moore and Lippman regarding positive development indicators, we replaced the ISEQ item “the activities really helped my learning of this topic” with “I asked myself questions to make sure I understood the class content,” thereby minimizing scoring bias in courses with limited in-class activities. Table 1 presents the questionnaire items used to assess multidimensional engagement: items 1, 3, and 5 measure behavioral, affective, and cognitive engagement, respectively, while items 2 and 4 reflect disengagement in behavior and emotion(Skinner, Kindermann and Furrer, 2009, Fuller et al., 2018). All items were rated on a five-point Likert scale (–2 to +2), and students completed the questionnaire immediately after each class to minimize recall bias. The questionnaire data collection process is illustrated in Figure 3. TABLE 1. Self-report items for measuring in-class engagement in online survey. Questions (please describe your engagement in the last class) Subscales 1. I paid attention in class. Behavioural 2. I pretended to participate in class but actually not. Behavioural (-) 3. I enjoyed learning new things in class. Emotional 4. I felt discouraged when we worked on something. Emotional (-) 5. I asked myself questions to make sure I understood the class content. Cognitive Note: (-) means the reversed score. During data processing, all reverse-coded items were first adjusted by inverting their scores, after which the raw scores were linearly transformed to a 1–5 scale. The arithmetic mean of the cognitive, affective, and behavioral scores was then calculated to yield the overall engagement ground truth for each class session. 2.2 DATA PREPROCESSING SETEASY 7 To accurately determine the start and end times of classroom sessions, we applied Information-Gain based Temporal Segmentation (IGTS)(Sadri, Ren and Salim, 2017). This method automatically differentiates between instructional and break periods based on student activity patterns, ensuring that all subsequent analyses are strictly confined to genuine classroom teaching intervals. To address prevalent motion artifacts and environmental noise in the physiological data, we implemented a multi-stage cleaning process: EDA signals were first smoothed using a 5-second median filter to suppress transient noise, then decomposed into tonic and phasic components via the cvxEDA algorithm. For BVP data, linear interpolation was employed to repair intervals with suboptimal heart rate quality. Following the removal of flat segments and artifact contamination, we validated a total of 331 high-quality, usable class sessions, which served as the foundational dataset for further analysis. 2.3 FEATURE-ENGAGEMENT ENGINEERING The central goal of this study is to construct effective features for precise classroom engagement prediction. Building on related work such as nGage(Gao et al., 2020, Zhou et al., 2023), we replicated and enhanced the multidimensional engagement prediction pipeline to fit our research context, resulting in the new v-Gage system module. From physiological signals, we extracted 36 features, including the amplitude and number of peaks in EDA signals, proportion of arousal, and HRV indices in both time (SDNN, RMSSD, pNN50) and frequency (LF/HF ratio, etc.) domains(Shaffer and Ginsberg, 2017). For behavioral patterns, we calculated the mean and variance of motion intensity from accelerometer data, and constructed synchrony features with both teachers and peers using Dynamic Time Warping (DTW) and Pearson correlation coefficients (8 features total). From classroom environmental data, we extracted 8 features—mean and peak values of CO₂, temperature, humidity, and noise during lessons—to quantify environmental impacts on students(De Dear and Brager, 2002). The visual behavior recognition stream further enriched the system with direct classroom behavioral features: for each class session, we detected the frequency and duration of the five most common behaviors, hand-raising-toteacher-response latency, and cumulative dozing time, resulting in 10 behavioral features per session. Altogether, this yielded a comprehensive feature set with 62 dimensions. We employed a LightGBM (Luxburg et al., 2018)regression model to model the relationship between the feature set and students’ classroom engagement. The LightGBM-based v-Gage automatically predicts three- 8 ZHIHAO XIE, HONGYE YANG AND SHIEN LIU dimensional engagement, offering a rapid, efficient, and quantifiable alternative to traditional subjective scoring. Model training followed a nested cross-validation scheme: the inner loop used three-fold cross-validation for hyperparameter tuning and feature selection, while the outer loop applied fivefold cross-validation (grouped by student ID) to ensure generalizability and prevent information leakage. 2.4 CONSTRUCTION OF THE STUDENT–SEAT UTILITY MATRIX To map multimodal predictions of individual classroom engagement to actual teaching space, we constructed a student –seat utility matrix that quantitatively evaluates the potential effects of every possible seating arrangement on classroom outcomes. First, we performed spatial calibration of the classroom using a single overhead 4K camera and ARUCO marker technology(Garrido-Jurado et al., 2014), enabling one-time calibration of the classroom layout and seat grid and assigning spatial coordinates and unique indices to each seat. During each class session, the visual model ran a realtime head tracking algorithm and employed the SORT multi-object tracking method(Bewley et al., 2016) to accurately map student IDs in every frame to their respective seats, allowing the system to automatically generate and store student–seat assignment sequences for each session. Based on these spatial calibration and seat-matching results, we aggregated the predicted engagement of each student for every seat over the preceding two weeks. Specifically, we calculated the average predicted engagement (𝐸 󰆽𝑖,𝑗) for each student –seat pair to reflect long-term trends in classroom performance, and computed the standard deviation (σ𝑗) for each seat to quantify the stability of engagement across different students. We then combined these metrics to define a comprehensive utility score: 𝑈𝑖,𝑗 = 0.8𝐸 󰆽𝑖,𝑗 + 0.2(1 𝜎𝑗 ⁄) (1) The weighting coefficients of 0.8 and 0.2 were selected based on sensitivity analysis and cross-validation, emphasizing the long-term performance of individual seats while appropriately considering the consistency and reliability of each seat’s engagement level. This utility matrix serves as a crucial input for the subsequent seat optimization model, quantifying the influence of classroom spatial configuration on overall classroom engagement. 2.5 SEAT OPTIMIZATION SOLVER Building on the established seat utility matrix, we rigorously modeled the seat assignment as an integer linear programming problem and solved it using the Google OR-Tools CP-SAT solver(Martinelli Tabajara and Y. Vardi, 2019) for SETEASY 9 efficient optimization. The decision variable 𝑥𝑖,𝑗 ∈{0,1} is set to 1 if student i is assigned to seat j, and 0 otherwise. The objective function is to maximize the total weighted utility across all students: max∑𝑤𝑖𝑖,𝑗 𝑈𝑖,𝑗 𝑥𝑖,𝑗 (2) Here, the weight wi is customizable by the teacher to address specific pedagogical needs —such as assigning higher weights to students with learning difficulties or attention deficits—enabling proactive support for these students through the optimization process. The optimization is subject to the following constraints: (1) Each student must be assigned to exactly one seat (student-seat uniqueness); (2) Each seat may be assigned to at most one student (seat-student exclusivity); (3) Vision and height constraints: students with impaired vision or shorter stature are prioritized for front-row seats with unobstructed views; (4) Social and collaboration constraints: avoid seating students who are prone to mutual distraction or negative interactions next to each other, while promoting proximity among students who collaborate effectively; (5) Accessibility and special needs: ensure that students with disabilities or special requirements are seated in accessible locations; (6) Teacher-specified reserved or restricted seating zones, providing teachers with necessary intervention capabilities. 2.6 MODEL DEPLOYMENT To evaluate the usability of the system in authentic classroom settings, all algorithm modules were deployed on the standard classroom computer used by teachers, with automatic data synchronization and analysis handled over the local area network(Xie et al., 2019, Liu et al., 2022). The classroom was equipped with a standard teaching computer (Intel Core i5, 16 GB RAM, NVIDIA RTX 3060, Ubuntu 20.04), serving as the central computation and storage node. All wearable physiological sensors, environmental monitors, and classroom cameras were connected to the same Wi-Fi LAN and configured to automatically upload collected data to a designated directory on this computer after each class, ensuring comprehensive and centralized data management. A unified Python 3.8 virtual environment was created on the deployment machine, with required dependencies —including PyTorch, OpenCV, LightGBM, Scikit-learn, and OR-Tools—installed sequentially. The final model weights and configuration files for the behavior recognition, engagement prediction, and CP-SAT seat optimization modules were 16 ZHIHAO XIE, HONGYE YANG AND SHIEN LIU 31 Jul 2025]. CASTELLUCCI, H.I., AREZES, P.M., and VIVIANI, C.A., 2010. Mismatch between classroom furniture and anthropometric measures in Chilean schools. Applied Ergonomics. 41 (4), pp. 563–568. CHOI, W.Y., YANG, J.H., and CHUNG, C.C., 2021. Data-Driven Object Vehicle Estimation by Radar Accuracy Modeling with Weighted Interpolation. Sensors. 21 (7), p. 2317. DE DEAR, R.J. and BRAGER, G.S., 2002. Thermal comfort in naturally ventilated buildings: revisions to ASHRAE Standard 55. Energy and Buildings. 34 (6), pp. 549–561. DI LASCIO, E., GASHI, S., KRASIC, D., and SANTINI, S., 2017. In-classroom selftracking for teachers and students: preliminary findings from a pilot study. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers [online]. Presented at the UbiComp ’17: The 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Maui Hawaii: ACM. pp. 865–870. Available from: https://dl.acm.org/doi/10.1145/3123024.3125504 [Accessed 31 Jul 2025]. FULLER, K.A., KARUNARATNE, N.S., NAIDU, S., EXINTARIS, B., SHORT, J.L., WOLCOTT, M.D., SINGLETON, S., and WHITE, P.J., 2018. Development of a selfreport instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem. PLOS ONE. 13 (10), p. e0205828. GAO, N., SHAO, W., RAHAMAN, M.S., and SALIM, F.D., 2020. n-Gage: Predicting inclass Emotional, Behavioural and Cognitive Engagement in the Wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 4 (3), pp. 1–26. GARRIDO-JURADO, S., MUÑOZ-SALINAS, R., MADRID-CUEVAS, F.J., and MARÍNJIMÉNEZ, M.J., 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition. 47 (6), pp. 2280–2292. GAY, G., 2002. Preparing for Culturally Responsive Teaching. Journal of Teacher Education. 53 (2), pp. 106–116. KAKOOEE, R. and DILLENBURGER, B., 2024. Reimagining space layout design through deep reinforcement learning. Journal of Computational Design and Engineering. 11 (3), pp. 43–55. KARIIPPANON, K.E., CLIFF, D.P., LANCASTER, S.J., OKELY, A.D., and PARRISH, A.- M., 2019. Flexible learning spaces facilitate interaction, collaboration and behavioural engagement in secondary school. PLOS ONE. 14 (10), p. e0223607. LADSON-BILLINGS, G., 2014. Culturally Relevant Pedagogy 2.0: a.k.a. the Remix. Harvard Educational Review. 84 (1), pp. 74–84. LIGGETT, R.S., 2000. Automated facilities layout: past, present and future. Automation in Construction. 9 (2), pp. 197–215. LIU, B., XING, W., ZENG, Y., and WU, Y., 2022. Linking cognitive processes and learning outcomes: The influence of cognitive presence on learning performance in MOOCs. British Journal of Educational Technology. 53 (5), pp. 1459–1477. LUXBURG, U. von, GUYON, I., BENGIO, S., WALLACH, H., FERGUS, R., VISHWANATHAN, S.V.N., GARNETT, R., and NEURAL INFORMATION PROCESSING SYSTEMS FOUNDATION, eds., 2018. Advances in neural information processing systems 30: 31st Annual Conference on Neural Information Processing Systems (NIPS 2017): Long Beach, California, USA, 4-9 December 2017. Presented at the NIPS, Red Hook, NY: Curran Associates, Inc. MÄKELÄ, T. and LEINONEN, T., 2021. Design Framework and Principles for Learning Environment Co-Design: Synthesis from Literature and Three Empirical Studies. Buildings. 11 (12), p. 581. SETEASY 17 MARTINELLI TABAJARA, L. and Y. VARDI, M., 2019. Partitioning Techniques in LTLf Synthesis. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence [online]. Presented at the Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19}, Macao, China: International Joint Conferences on Artificial Intelligence Organization. pp. 5599–5606. Available from: https://www.ijcai.org/proceedings/2019/777 [Accessed 31 Jul 2025]. MELLER, R.D. and GAU, K.-Y., 1996. The facility layout problem: Recent and emerging trends and perspectives. Journal of Manufacturing Systems. 15 (5), pp. 351–366. MORENO-MARCOS, P.M., ALARIO-HOYOS, C., MUNOZ-MERINO, P.J., and KLOOS, C.D., 2019. Prediction in MOOCs: A Review and Future Research Directions. IEEE Transactions on Learning Technologies. 12 (3), pp. 384–401. OECD, 2013. Innovative Learning Environments [online]. OECD. Available from: https://www.oecd.org/en/publications/innovative-learning-environments_9789264203488en.html [Accessed 31 Jul 2025]. PEKRUN, R. and LINNENBRINK-GARCIA, L., 2012. Academic Emotions and Student Engagement. In: S.L. CHRISTENSON, A.L. RESCHLY and C. WYLIE, eds. Handbook of Research on Student Engagement [online]. Boston, MA: Springer US. pp. 259–282. Available from: https://link.springer.com/10.1007/978-1-4614-2018-7_12 [Accessed 1 Aug 2025]. PEPONIS, J., BAFNA, S., BAJAJ, R., BROMBERG, J., CONGDON, C., RASHID, M., WARMELS, S., YAN ZHANG, and ZIMRING, C., 2007. Designing Space to Support Knowledge Work. Environment and Behavior. 39 (6), pp. 815–840. SADRI, A., REN, Y., and SALIM, F.D., 2017. Information gain-based metric for recognizing transitions in human activities. Pervasive and Mobile Computing. 38, pp. 92–109. SHAFFER, F. and GINSBERG, J.P., 2017. An Overview of Heart Rate Variability Metrics and Norms. Frontiers in Public Health. 5, p. 258. SHIN-IKE, K. and IIMA, H., 2016. A Method Based on a Genetic Algorithm for Classroom Seating Arrangements. SICE Journal of Control, Measurement, and System Integration. 9 (3), pp. 144–150. SKINNER, E.A., KINDERMANN, T.A., and FURRER, C.J., 2009. A Motivational Perspective on Engagement and Disaffection: Conceptualization and Assessment of Children’s Behavioral and Emotional Participation in Academic Activities in the Classroom. Educational and Psychological Measurement. 69 (3), pp. 493–525. XIE, H., CHU, H.-C., HWANG, G.-J., and WANG, C.-C., 2019. Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Computers & Education. 140, p. 103599. YANG, F., WANG, T., and WANG, X., 2023. Student Classroom Behavior Detection based on YOLOv7-BRA and Multi-Model Fusion. [online]. Available from: https://arxiv.org/abs/2305.07825 [Accessed 31 Jul 2025]. ZHOU, H., JIANG, F., SI, J., XIONG, L., and LU, H., 2023. Stuart: Individualized Classroom Observation of Students with Automatic Behavior Recognition And Tracking. In: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) [online]. Presented at the ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece: IEEE. pp. 1–5. Available from: https://ieeexplore.ieee.org/document/10094982/ [Accessed 31 Jul 2025].