Context-Dependent Labeling of Improvisational Piano Performance in Early Childhood Education and Therapy

Context-Dependent Labeling of Improvisational Piano Performance in Early Childhood Education and Therapy Risa Kobayashi1,2[0009→0007→1625→2316],KanaeSuzuki 1[0009→0008→5690→630X], Nami Iino1,4[0000→0001→7806→7131], Ikki Ohmukai3[0000→0002→3276→3753],and Hiroko Terasawa1[0009→0000→3427→0031],MasakiMatsubara 1[0000→0003→1950→683X] 1University of Tsukuba 2Seitoku University 3The University of Tokyo 4RIKEN Center for Advanced Intelligence Project (AIP) [email protected] Abstract. In early childhood education and therapeutic settings, musical performances are often improvised to meet children’s developmental needs. While such improvisation plays a vital role in supporting communication, emotion, and engagement, it has rarely been systematically described or analyzed. This study introduces a method for documenting and interpreting improvised piano performances used by educators and therapists, with the goal of making context-sensitive musical expertise more visible and transferable. We propose a dual-layer labeling system that captures both the type of musical activity (e.g., singing, movement, instrument, listening) and the performer’s real-time intention (e.g., learning, enjoyment, interaction). This structure is combined with two simple acoustic indicators—horizontal and vertical note density—to quantify the texture of performance. Data from 41 musical segments were collected across daycare, kindergarten, and therapy settings. Results show that even when the same song is used, performance characteristics vary measurably depending on children’s states and pedagogical aims. Comparative analysis with Or!and ISO frameworks suggests that the proposed approach adds finer granularity to practice-based analysis. Keywords: Improvised Performance ·Early Childhood Education and Music Therapy ·Labeling and Visualization. 1Introduction For children whose language functions are still developing, music is considered easier to understand and more impactful than verbal communication [10]. EducaAll rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 647 R. Kobayashi et al. tors use music and musical elements to communicate and guide children according to their conditions and situations [20]. It has been shown that synchronizing music with a child’s mood enhances empathy [18], and aligning rhythm with bodily movement improves interpersonal synchrony and cooperation [14,22]. In recent years, kindergartens and daycare centers have increasingly included children with disabilities, and the e!ectiveness of musical activities from a music therapy perspective has been reported [12]. In such settings, it is possible that characteristic performance patterns and changes emerge depending on the performer’s goals and perspectives. However, extracting these commonalities is di"cult, and such practical knowledge has not been made explicit [19]. As a result, educational and therapeutic practices have largely depended on individual experience and intuition, and the question of what kind of performance is truly e!ective remains unclear [8]. Moreover, many early childhood educators have limited experience with piano performance, and the number of individuals who can flexibly adapt their playing to the situation is declining [9,23]. Meanwhile, growing interest in the application of AI in music education [5,24] has made it increasingly important to understand how dynamic adaptability in musical performance is constituted, with a view toward future implementation of improvisational AI systems [15]. In response to these challenges, this study focuses on improvised piano performances in educational and therapeutic settings. We aim to construct a framework for identifying and analyzing di!erent performances by using “context labels” and “intention labels” that represent children’s states and the purpose of the activity, along with simple acoustic features that capture performance di!erences, as illustrated in Fig. 1. One advantage of this study is that the first author has professional experience in piano performance, enabling expert analysis of musical expression and change. Furthermore, the comparative investigation of improvisational performances from the perspective of performer knowledge—across the contrasting practical contexts of education and therapy—represents a novel contribution to the field. Through this analysis, we aim to visualize the characteristics and changes in performance that arise in practice, thereby laying the groundwork for future automated understanding and classification of musical improvisation. Fig. 1. Overview of the proposed process Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 648 Context-Dependent Piano Performance in Education and Therapy 2RelatedWork 2.1 Improvisation Research: From Jazz to Educational Contexts Improvisation has long been studied in musicology and performance studies, with influential works like Berliner’s Thinking in Jazz [2] analyzing jazz improvisation as sophisticated real-time musical reasoning. Pressing [17] contributed a cognitive model emphasizing mental schemata and decision spaces, while Borgo [3] explored collective creativity in improvised settings. However, these studies primarily address contexts where musical complexity and aesthetic values are central. In contrast, improvisation in early childhood education and therapy serves fundamentally di!erent purposes: supporting developmental progress, emotional regulation, and social engagement. These nonmusical goals require performers to prioritize adaptability and pedagogical intent over musical virtuosity. Despite the importance of such practices, there is a notable gap regarding how improvisation functions in educational and therapeutic domains. Few studies have attempted to model improvisational behavior contingent on real-time assessments of child behavior and situated learning objectives. 2.2 Labeling and Analysis in Music Performance Various frameworks label musical behavior, from jazz chord functions (ii–V–I progressions, modal interchange) to developmental stages in education. The Or! Schulwerk approach emphasizes exploration, imitation, and creation [7], while music therapy employs the ISO Principle to adjust musical features—tempo, tonality—to match client states [1]. These systems provide useful conceptual sca!olds but are typically prescriptive or retrospective, focusing on musical structure or generalized developmental stages without accounting for performers’ micro-level decisions in response to specific, observable behaviors. 2.3 Performance Adaptation in Educational Settings Music therapy research shows that improvisational interaction unfolds through engagement and response stages. Streeter [21] described transitions from therapistled unidirectional performance to client-led tempo leadership through intermediate stages of tempo establishment and rhythmic imitation. In music education, Nishizono [16] described musical enjoyment developing through social interaction and musical skill acquisition, progressing from aligning rhythms to sharing expressive variations. These studies suggest that musical interaction develops continuously and in multiple layers, providing a conceptual basis for understanding children’s behavior and performers’ adaptive improvisation in this study. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 649 R. Kobayashi et al. 3Methods 3.1 Research Sites and Data Collection As summarized in Table 1, this study was conducted in three practical settings with di!erent backgrounds: a daycare center, a kindergarten, and a music therapy clinic. Three musical sessions were recorded in each setting, resulting in a total of nine video-recorded sessions. From these recordings, 10 performances were extracted from the daycare center, 14 from the kindergarten, and 17 from the music therapy sessions, yielding 41 performance segments in total for analysis. Some songs were performed multiple times across di!erent sessions or contextual situations. All musical activities were accompanied by piano, and no restrictions were placed on repertoire or activity type, in order to capture natural and improvisatory practices. In the daycare and kindergarten settings, the researcher conducted participant observation. In contrast, during the music therapy sessions, the researcher’s presence was found to potentially distract the child in the first session. Therefore, from the second session onward, a non-participant observation approach was adopted. All data collection procedures were approved by the ethics review committee of the authors’ institution. 3.2 Proposal of systematized labeling method Based on observations of the recorded videos and interviews with the performers, we devised a tentative labeling system to describe children’s behavior, states, and the performer’s intentions. To incorporate theoretical perspectives on the goals and stages of participation in musical activity, we referred to prior studies [21, 16]. The conceptual basis of the labeling structure is illustrated by the following statement from one of the performers: Table 1. Overview of Collected Data Location Description Daycare center Morning circle time and music activities with 2to 4-year-old children. The 2-year-old class had about 10 children, and the 3and 4-year-old classes had 17–23 children each. Performances were given by three homeroom teachers and the principal. Kindergarten Music activities in classes of 4and 5-year-old children, each class having approximately 20–25 children. The performer was a teacher who had graduated from a music college. Music therapy One-on-one sessions with an 8-year-old child attending a special support school and their mother. The performer was a certified music therapist. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 650 Context-Dependent Piano Performance in Education and Therapy “I don’t really use scores. If the children seem able to sing with the accompaniment, I go with it. If not, I play the melody with my right hand. I change how I play depending on how the children seem at that moment.” This remark highlights how performers adapt their playing in real time by attending to both the child’s condition and their own musical intention. In this study, we designed a two-layer labeling system that combines upperlevel categories representing the type of musical activity (Table 2) with lowerlevel categories representing the child’s state and the performer’s intention (Table 3). Examples of concrete label combinations include: – S-LRN (Singing – Learn):Performancesintendedtosupportchildrenas they try to learn lyrics or melodies. – I-ENJ (Instrument – Enjoy):Performancesthatencouragespontaneous interaction with instruments for enjoyment. – L-OBS (Listening – Observation):Performancesthataimtoengage children who are not actively participating but are listening attentively. Each individual category alone may be too abstract for analysis; however, combining them allows for a more concrete understanding of why a specific song was selected and how the performer’s intention responded to the child’s behavior or developmental stage. Labeling was conducted by two researchers, including the first author. When the child’s activity or condition changed during a single performance, labels were assigned based on the most dominant feature across the entire segment. Final labeling decisions were confirmed through discussion with the performer. 3.3 Visualization Method In music analysis, the concept of texture has been discussed from theoretical and perceptual perspectives [11,6]. In an attempt to quantitatively treat texture in piano scores, Couturier et al. [4] proposed a method to compare and visualize stylistic and structural differences between classical music pieces, focusing on vertical density (number of simultaneous notes) and horizontal density (density of sounds per time). Table 2. Categories of Musical Activities Initials Category SSinging IInstrumentPlaying MMovement/PhysicalExpression L Listening Table 3. Label Abbreviations and Their Meanings Abbreviation Meaning LRN Learn ENJ Enjoy INT Interact EXP Express IMIT Imitate OBS Observe Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 651 R. Kobayashi et al. In this study, we also refer to the work of Kawase et al. [13] in a way that corresponds to the concepts of vertical and horizontal density, and attempt to visually capture the structural characteristics of left-hand accompaniment using the following indicators: ANNM (Average Number of Notes per Measure):ANNMisanindicator that represents the number of note-onset events occurring within a single measure. A larger value indicates that more notes are played within the measure, reflecting a performance with higher rhythmic density. ANB (Average Notes per Beat):ANBisanindicatorthatrepresentsthe average number of notes sounding simultaneously per beat within a measure. A larger value indicates that more notes are included in the chords, reflecting a performance with higher harmonic density. These indicators were calculated either for each measure or across the entire performance as averages, and were used as supplementary information to visually identify performance tendencies and local changes. This allows us, for example, to visually capture how the performance changes in response to situational changes during a session. 4ResultsandDiscussion 4.1 Distribution of Labels and Field-Specific Characteristics Based on the labeling framework presented in the previous section, each performance instance was assigned a label (see Table 4). This subsection provides an overview of the resulting distribution. Although the theoretical number of label combinations is 24 (4 upper-level categories ×6lower-levellabels),only11typesoflabelswereactuallyobserved in the collected data. Table 5 summarizes the label frequencies by field. It should be noted that this analysis is based solely on the dataset collected in this study and is not intended to generalize beyond it. –In the daycare center, labels categorized under ENJ (Enjoy) and INT (Interaction) appeared with a frequency of 40% each. –In the kindergarten data, S-EXP (42.9%) and M-INT (28.6%) were dominant, reflecting expressive singing and physical interaction activities. –In music therapy sessions, OBS (29.4%) and IMIT (17.6%) were assigned. These labels correspond to the initial stages of musical interaction described by [21]. 4.2 Relationship between Labels and Distributions Next, we examined the relationship between the collected performance data and the labels classified based on observation. Fig. 2 presents a scatter plot of the Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 652 Context-Dependent Piano Performance in Education and Therapy Table 4. Excerpt of Performance Data by Location and Song Title (12 out of 41 pieces) Location Song Title (English / Japanese Romanization)ANNMANB Label Daycare center Omocha no Cha-Cha-Cha 2.00 1.00 I-ENJ Daycare center Mamemaki 4.25 1.44 S-INT Daycare center The Bear Song 5.00 1.25 S-ENJ Kindergarten Jankenpon (1st performance)*1 5.30 1.16 S-LRN Kindergarten Jankenpon (2nd performance)*1 6.81 1.30 S-INT Kindergarten Niji 3.00 1.15 S-EXP Kindergarten Chocolate 3.38 2.44 M-INT Music therapy Do-Re-Mi 3.72 1.79 I-INT Music therapy Sanpo (1st performance)*2 3.68 1.53 L-OBS Music therapy Sanpo (2nd performance)*2 3.76 2.12 M-ENJ Music therapy It’s a Small World 5.68 1.83 M-IMIT Performances marked with *1 and *2 are introduced as case studies in Section 4.3. Table 5. Number of Performances by Label and Location Location Total ILRN IENJ IINT SLRN SENJ SINT SEXP MINT MENJ MIMIT LOBS Daycare center 10 2 2 1 2 2 1 Kindergarten 14 1 2 1 6 4 Music therapy 17 3 3 2 1 1 1 3 5 ANNM and ANB values for each piece, color-coded by activity type (Movement, Listening, Instruments, Singing). The results showed that Movement (→)tended to have overall higher ANB values compared to the other activity types. Listening (↭)exhibitedarelatively cohesive distribution, with ANNM values concentrated around 3 to 4 and ANB values around 1.5 to 2. In contrast, Instruments (+)and Singing (•)both showed large variability, and no clear tendencies were observed based on activity type alone. Therefore, in Fig. 2, we reclassified the performances for Instruments and Singing, where no clear patterns had emerged, focusing instead on the performer’s intended aims (Enjoy, Express, Interaction, Learn). We then compared the distributions of ANNM and ANB based on this reclassification (Fig. 3). As a result, performances classified as Enjoy and Express showed a wide range of both ANNM and ANB, suggesting that they reflected the atmosphere and expressive qualities of the pieces. In contrast, performances classified as Interaction were relatively concentrated within ANNM values of 1.5 to 4.5 and ANB values of 1.0 to 2.5. Furthermore, performances classified as Learn had ANB values falling between 1.0 and 1.5, indicating that they primarily involved single-note accompaniments. This likely reflects a focus on making the melody line more audible, with the intent of aiding song and lyric learning. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 653 R. Kobayashi et al. These findings indicate that the relationship between performance data and labels becomes more clearly visible and interpretable not only in terms of activity types (Fig. 2), but also through classification based on the performer’s intent and objectives (Fig. 3). Fig. 2. Scatter plot of ANNM and ANB by activity type. Movement shows higher ANB, Listening shows a concentrated distribution, while Instruments and Singing show large variability. Fig. 3. Reclassified Instruments and Singing by performer intention. Enjoy/Express show wide distributions, Interaction is relatively concentrated, and Learn is characterized by single-note accompaniments. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 654 Context-Dependent Piano Performance in Education and Therapy 4.3 Visualization of Performance Variations in the Same Song This section examines how variations in performance can be visualized when the same performer plays the same piece under di!erent contextual conditions and child states. By using time-series graphs that show indicator values on a permeasure basis, di!erences between two performances can be more intuitively and visually understood. Specifically, the following di!erences were observed: Di!erences in singing acquisition (Fig. 4): In Performance 1, both ANNM and ANB values were low in the first half, indicating a simplified accompaniment. This scene was labeled as S-LRN, in which the child was in the process of acquiring the lyrics and melody. Interview data also confirmed that the performer intentionally simplified the left-hand accompaniment to clearly support the child’s singing. In contrast, Performance 2, labeled S-INT, showed segments with denser accompaniment compared to Performance 1, reflecting a more natural and flowing performance style appropriate for a post-acquisition context. These di!erences could be clearly identified in the time-series graph. Di!erences in activity type and level of engagement (Fig. 5): In a music therapy session, we compared a scene with low participation (L-OBS) and another in which rhythmic foot-tapping emerged midway through the song (MINT, Performance 2). Around the 8th measure, where the child began to engage in coordinated movement, the ANB value increased, indicating a shift toward a denser and more harmonically supportive accompaniment. This change aligned with the onset of physical activity and was visually apparent in the graph. These results suggest that the use of ANNM and ANB, calculated as measurelevel averages, allows for the visualization of fine-grained performance variations. Furthermore, the findings indicate the importance of capturing such micro-level changes when attempting to detect immediate, responsive adaptations to the child’s state. 4.4 Comparison with Existing Labeling Frameworks To assess the applicability and distinctiveness of our dual-layer labeling system, we compared it with two widely recognized frameworks in early childhood music practice: the Or!Schulwerk approach and the ISO Principle in music therapy. Table 6 summarizes this correspondence, highlighting how our labels map onto these existing models and where they o!er additional granularity. The Or!Schulwerk method conceptualizes musical learning through stages such as exploration, imitation, and creation. While our labels like I-ENJ and S-LRN align broadly with these stages, the proposed system adds specificity by explicitly capturing performer intention in situ. The ISO Principle emphasizes musical synchrony with the child’s current state. Our labels such as L-OBS and M-INT operationalize this principle by encoding observable behaviors alongside performer response strategy. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 655