Proposal of a Singing Instruction Support System with an Inhalation Restriction Function to Demonstrate the Difference in Physical Ability between a Teacher and Student in Online Lessons

Proposal of a Singing Instruction Support System with an Inhalation Restriction Function to Demonstrate the Difference in Physical Ability between a Teacher and Student in Online Lessons Hiroya Miura1,AkiyoObara 2,HirokiWatanabe 2, Yoshinari Takegawa2,andKeijiHirata 2 1RIKEN Center for Advanced Intelligence Project 2Future University Hakodate [email protected] Abstract. In the teaching of singing, breathing control is an important component, the observation of which has conventionally required ateacher’slongexperienceandrefinedperception.However,inonline lessons where visual information is limited, there is the problem that physical differences are hard to perceive and it is difficult to obtain the information necessary to provide instruction. In this research, which uses aresidualrespiratoryvolumevisualizationsystembasedonpreviousresearch, we verify the system’s usefulness for supporting teaching, as well as implementing a new function to restrict inhalation volume based on a student’s lung capacity, and evaluating the effects of this function. Experimental results implied that the proposed system is useful as a tool to enable teachers to recognize the physical difference between themselves and their students. Keywords: Teaching Support ·Singing Lecture ·Visualization. 1Introduction The promulgation of online lessons during the Covid-19 pandemic led to the verification of the effectiveness of learning and teaching in such lessons in comparison to face-to-face lessons. It has been found that online lessons have a higher degree of student satisfaction than face-to-face lessons, and have the effect of increasing autonomy and concentration [1–4]. In addition, it has been pointed out that, by enabling remote lessons, online lessons can potentially help solve the problems of a lack of teachers and mismatches between students and teachers [5,6]. From this, we consider that online lessons are not simply a replacement for face-to-face lessons, and will continue to be chosen long after the end of the pandemic. All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 33 Miura et al. On the other hand, when compared to face-to-face lessons, in online lessons it is difficult to obtain information needed for instruction [1]. Stabilized breathing is essential for singing, but judging whether a student’s breathing process, which takes place internally, is appropriate requires minute observation and long experience. However, in an online lesson, the teacher checks the student’s performance on a monitor, making it difficult to observe body movements when compared to a face-to-face lesson. Furthermore, it is difficult to grasp a student’s actual height and chest size. Accordingly, it is also difficult to surmise what vocal volume and breathing duration a student is capable of. The aim of this research is to construct a system to provide teaching support in online lessons by visualizing residual respiratory volume during singing. Here, residual respiratory volume refers to the amount of breath remaining when a person sings. We confer on the teacher a fresh viewpoint for instruction, by providing information about an individual student’s breathing, and verify whether this contributes to teaching support. Also, we construct a function (residual respiratory volume display function) that aids the comprehension of physical difference, by comparing the respective residual respiratory volumes of the student and the teacher. In particular, we evaluate the function that restricts the volume of the teacher’s inhalation (inhalation restriction function). This function enables the teacher to share the sensation of the student’s breathing and experience the physical difference between them. 2RelatedResearch Lã et al. [7] determined that real-time visual feedback of performance information is effective at enabling more efficient mastery of singing. Several mastery support systems that display acoustic information during singing have actually been proposed [8,9]; more recent interfaces explicitly incorporate breath guidance and multi-sensor analyses for pedagogical feedback [10, 11]. However, there are few cases that focus on breathing during singing. In general, to measure breathing during singing the following conditions must be met: (1) measurement can be carried out while the singer is standing still, (2) measurement must be carried out safely, (3) measurement equipment is small and easy to use, (4) measurement has a minimal effect on the movements involved in breathing. As a means that satisfies these requirements, there is the method of using a sensor to measure the body movements that accompany singing. In this method, the contraction of the chest or abdomen, which accompanies breathing, is measured using a device such as an inductor [12] or stretch/positional sensors with learned mappings from thoracoabdominal motion to volume [13,14]. The waveform of this body movement and the waveform of breathing have been shown to have a strong relationship to the waveform measured by a flow meter and to objective respiratory patterns in trained singers [15]. Foundational work further established the decomposition of ribcage and abdominal contributions to breathing volume, which underpins modern respiratory inductance plethysmography and related approaches [16]. In this research, we decided to target the Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 34 Singing Instruction Support System with an Inhalation Restriction Function Stretch Sensor M5stack System Configuration Installation Example Fig.1: Wearing Breathing Measurement Equipment teacher and display residual respiratory volume to support teaching. We verify whether this is effective teaching support by presenting the residual respiratory volume to a teacher who is providing singing instruction. In addition, we implement a new function to impose a restriction on the teacher in accordance with the difference in the residual respiratory volumes of the teacher and the student. 3ProposedSystemFunctions In this research, our goal is to design and implement a teaching support system that aims, in particular, to enable mastery of breathing in singing. Specifically, this is realized by displaying residual respiratory volume, obtained from information gathered by sensors attached to the body. 3.1 Residual respiratory volume estimation function The structure of the breathing measurement system and an example of wearing the device are shown in Figure 1. Stretch sensors are used to measure chest movement. A stretch sensor is a variable resistor that expands and contracts like rubber. The sensors, which are fixed onto belts that can be attached and removed using buckles, are attached to two places: the chest and abdomen. The sensor values are sent to the master PC via an M5Stack, a compact, all-in-one development board designed for rapid prototyping and embedded applications. M5Stack and the master PC are connected by Bluetooth. An outline of the residual respiratory volume estimation part of the system is presented in Figure 2. Reception of the data from the breathing measurement system, and the implementation of the display screen showing graphs etc., was carried out in Processing. Estimation of residual respiratory volume was implemented in Python. The sensor data received from the breathing measurement system is sent to the estimation part via consecutive text files. As preprocessing for estimation, the sensor data offset is adjusted. The estimation part estimates residual respiratory volume from the data received from the sensors on the chest and abdomen. The KNN (K Nearest Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 35 Miura et al. Respiration Measurement Offset Calculation Writing Reception Reading Predict (KNN) Writing Reading ・ Processing Display Section Reception Section Estimation Section Offset.txt rawData.txt predictIO.txt train.csv Processing Processing Python UI screen Fig.2: Overview of Residual Respiratory Volume Estimation Part Neighbour) method is used for estimation. The KNN method selects klearning data that are close to input data with unknown object variables, and obtains the average value of object variables of the learning data to estimate the object variable of the input data. In this research, the instantaneous sensor values from the chest and abdomen are feature values and the residual respiratory volume is the object variable. For the kparameter, we cross validated the learning data for each experiment participant and selected the value with the smallest error. Here, error refers to the RMSE (Root Mean Squared Error). The RMSE is obtained by the following formula: RMSE =v u u t1 n n X i=1 (yiˆyi)2 –n:No.ofsamples –yi:Estimatedvalue –ˆyi:Actualvalue To estimate residual respiratory volume we correlated the sensor values and residual respiratory volume and constructed datasets. A dataset is constructed each time the system user puts on the sensor belt. The user breathes into a spirometer, which measures lung capacity, and the sensor values during exhalation are recorded. In addition, the fluctuating scale on the spirometer is captured with a video camera. The measurement range of the spirometer is between 1000mL and 7000mL, in degrees of 100mL. After measurement, we correlated the sensor values and residual respiratory volume for every 100mL. 3.2 Residual respiratory volume display function This function displays the system user’s residual respiratory volume in real time. The left side of Figure 3 is an overview of the basic screen. The basic screen Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 36 Singing Instruction Support System with an Inhalation Restriction Function waveform of residual respiratory volume residual respiratory (%) expressing residual respiratory with color musical score Fig.3: Screenshot of Residual Respiratory Volume Display Function (Left) and Screenshot of Inhalation Restriction Function (Right) comprises a part that plots the waveform of residual respiratory volume, a part that displays residual respiratory volume by percentage and color change, and a part that displays the musical score. The waveform and the score are displayed moving across the screen from right to left, while the percentage and the color change along with the latest residual respiratory volume. This makes it possible to see the extent to which the singer is using their breath at each point on the score. 3.3 Inhalation volume restriction function Avisual-basedcontrolmethodthatmakestheteacheradjusttheirinhalation volume while comparing it with the student’s residual respiratory volume enables restriction of breathing without changing the tempo or volume of singing. By looking at the target inhalation volume and the state of their own inhalation, on the screen, the teacher can understand the extent to which they must restrict inhalation in order to sing with a residual respiratory volume equal to that of the student. To restrict the teacher’s breathing, we consider a method of limiting the maximum value of inhalation volume to that of the student’s lung capacity. However, it is difficult to restrict inhalation just by one’s own sensation. Therefore, we implemented a function to display a black-colored belt on the basic screen. The right side of Figure 3 presents an example of the display screen when this function is in use. The ratio of the student’s lung capacity to the teacher’s lung capacity is taken as the standard for the maximum inhalation rate, and the area on the graph above the standard is colored black. For example, if the teacher’s lung capacity is 5000mL and the student’s lung capacity is 2500mL, the part of the graph above the 50% mark will be colored black. If the teacher’s inhalation volume exceeds the standard, the teacher is notified by a warning sound. By breathing in such awayasnottoexceedthestandard,theteachercanartificiallyexperiencehow it feels to perform with the residual respiratory volume of the student. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 37 Miura et al. Table 1: Experiment Participant Data ID 123456789 Gender mfmmmfmf f Lung capacity (mL) 4400 2700 3550 5700 5100 2700 6500 1850 1500 Error (mL) 69.9 56.7 134.4 44.0 46.5 49.1 73.9 162.1 75.0 k224222222 When singing, there are cases in which the timing means that one cannot inhale a sufficient amount in accordance with one’s lung capacity. However, this is due to a lack of proficiency and not to a physical limitation. Accordingly, it is thought that, rather than altering the standard with each breath, taking the student’s lung capacity as the maximum inhalation volume permitted to the teacher is appropriate for the purpose of this function. 4 Evaluation of the Inhalation Volume Restriction Function The inhalation volume restriction function described in section 3.3 was evaluated in terms of whether breathing can be restricted appropriately. The experiment participants were presented with multiple standards of permitted maximum inhalation volume, and asked to adjust their breathing accordingly. The function was evaluated by comparing the difference between the standard and the actual inhalation volume in the cases of using and not using the function. 4.1 Experiment conditions and procedure Nine university students participated in this experiment. Their information is presented in Table 1. As experiment procedure, first, the lung capacity of the participants was measured once in order to construct learning data. Next, the target value of maximum inhalation volume was presented and breathing was measured. The measurement process was as follows. First, participants practiced breathing several times. Afterwards, while listening to a metronome set to 60 BPM, participants inhaled once every 4 beats, a total of 5 times. There were 6targetvalues:50%,60%,70%,80%,90%,and100%.Eachtargetvaluewas tested with 5 inhalations per participant, and this 6-value block was performed under two conditions (with and without the function), yielding 30 inhalations per condition (60 total per participant). Regarding the order in which the target values were presented, 100% was always presented first, after which the remaining 5valueswerepresentedinrandomorder. After measurement for the six target values had been carried out with the participants not using the inhalation volume restriction function, measurement was carried out in the same manner with the participants using the function. The Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 38 Singing Instruction Support System with an Inhalation Restriction Function order of the target values was the same as in the case of not using the function. Analysis was carried out by the following process. First, the peak value of the five inhalations for each standard was obtained. The absolute percentage error was calculated for each peak. The absolute percentage error is calculated with the following formula. =|Targetypeak VC ⇥100| –Target:Standardvalue(%) –ypeak:Estimatedvalueofpeak(mL) –VC:Lungcapacity(mL) When the number of samples is n,theMAPE(MeanAbsolutePercentage Error) is calculated with the following formula. MAPE =1 n n X i=1 i 4.2 Results and consideration We obtained the MAPE of inhalation volume up to the presented standard value, and compared the case of using the inhalation volume restriction function and the case of not using the function. The resulting values were 18.19% without the function, and 14.24% with the function. The number of samples was 270. Regarding the MAPE in the cases of using and not using the function, when atwo-sidedttestwascarriedout,thet-statisticwas3.49,p<.01, indicating asignificantdifference.Accordingly,itcanbesaidthatbreathingusingthe inhalation restriction function decreases the error between the actual inhalation volume and the target value. The percentage of samples in which the experiment participant’s breathing was below the target value was 56.7% in the case of without function and 74.1% in the case of with function. When a chi-squared test was carried out, there was shown to be a significant difference (2=17.31, p<.01). In addition, we extracted only samples in which breathing surpassed the target value, and calculated the MAPE of those samples. The resulting values were 18.31 (n=117) for without function and 10.34 (n=70) for with function. When a two-sided Welch’s ttestwascarriedout,thetstatisticwas4.46,p<.01, showing a significant difference. Accordingly, it can be said that the inhalation volume restriction function is effective at inducing the user to limit their inhalation volume so as not to exceed the target value. From this, it can be understood that the inhalation volume restriction function appropriately imposes a restriction on the teacher in correspondence with the student’s lung capacity. 5VerificationoftheTeachingSupportEffectofthe Inhalation Volume Restriction Function Auserstudywasconductedtoverifytheteaching-supporteffectoftheinhalation volume restriction function. The experiment recreated an online lesson and Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 39 Miura et al. Table 2: Participant Data and Presented Maximum Inhalation Rate Participant 12-1 2-2 Gender fm Teaching experience (years) 14 23 Lung capacity (mL) 3300 4700 Inhalation rate (%) 82.7 58.1 49.1 UI Screen Subject Video of a Student Singing Fig.4: Experiment Conditions the participants observed the residual respiratory volume of singing students, using pre-recorded video of the students singing and the residual respiratory volume visualization function. In addition, the participants wore sensor bands and actually used the inhalation volume restriction function. To verify the teachingsupport effect of this function, semi-structured interviews were conducted, in which participants answered questions about what they noticed regarding both the physical difference between themselves and the students and methods of instruction. 5.1 Experiment conditions The participants in this experiment were two educators with sufficient experience of teaching singing. The participant data is presented in Table 2. The experiment environment is presented in Figure 4. Two monitors were placed in front of the participants. The video of students singing was shown on the left monitor and the inhalation volume restriction function display screen was shown on the right monitor, as seen from the participant’s point of view. The monitors were connected to a master PC by HDMI cables. 5.2 Experiment procedure First, the participants’ lung capacity was measured once in order to construct adataset.Next,toenablecomparison,theparticipantswatchedthevideos recorded in advance. The screen displayed video of a student singing, and the residual respiratory volume of the student only. The participants watched the same video three times. Afterwards, a semi-structured interview was conducted. Following that, the participants sang while looking at the video of the student and the inhalation volume restriction function screen. The maximum rates of inhalation presented to the participants are shown in Table 2. As the experiment was conducted with two students, there are two standard rates, one for each student. The participants were instructed to breath in such a way as not to exceed the standard. After they had sung three times, the semi-structured interviews were conducted. The pre-prepared questions used in the interview were as follows: “Do you feel a physical difference between this student and yourself?” and “What kind of instruction can you give this student?” Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 40 Singing Instruction Support System with an Inhalation Restriction Function 5.3 Results When singing using the function that restricts inhalation volume, the experiment participants made observations, listed below, from the viewpoints of (1) introducing awareness that is different from usual, (2) physical difference between oneself and the student, (3) usefulness regarding teaching support, and (4) usefulness regarding online lessons. Note that the text in brackets within the quotations is context added by the authors. Introducing awareness that is different from usual When a teacher sang with restricted breathing, it was found that they became aware of different aspects from usual and made new realizations. For example, comments such as “Payingattention(tohowmuchIinhaled)mademerealizethatIprobably need to use things like the surrounding muscles more...I usually depend on breathing, but because I couldn’t do that I felt like I was using other parts of my body.” (Participant 1) showed that the participants were contriving new ways to achieve the best possible vocalization under the breathing restrictions. This reveals how, despite the fact that the restriction actually imposed by the inhalation volume restriction function is only on the amount of inhalation, singing under this restriction increases the burden on the muscles used for vocalization as well as the accuracy with which those muscles are used. Thus it can be said that this enables a teacher to recognize anew exactly which parts of the body are being used when singing. In addition, the following comment was obtained regarding the display of the teacher’s own waveform. “I realized that I don’t use much breath, as I had a lot to spare. I was thinking I’d try to inhale once every quarter rest (but I was able to sing without inhaling that often).” (Participant 2) Looking at their own waveform enabled this teacher to recognize the gap between the breathing they had supposed they were doing before and the way they were actually breathing. This resulted in the teacher changing their behavior, limiting their inhalation volume in accordance with the fact that they had a surplus residual respiratory volume. This suggests that using the system provided new realizations regarding a teacher’s own performance and the way they use their own body. Physical difference between oneself and the student The experiment showed that singing under restriction made the teachers feel the physical difference between themselves and the students. The teachers not only gained an objective understanding of the difference in numerical values, but, by singing in astateofrestriction,alsoactuallyexperiencedthestrainfeltbythestudents. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 41