Supporting Narrative Comprehension in Programmatic Music through Music and Light

Supporting Narrative Comprehension in Programmatic Music through Music and Light Anna-Maria Christodoulou1,HughAlexandervonArnim 1,andAlexander Refsum Jensenius1 RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, Department of Musicology, University of Oslo Abstract. Programmatic music, such as Tchaikovsky’s Overture Romeo and Juliet, relies on the audience’s ability to associate musical motifs with narrative elements. This is a demanding task for less experienced listeners, particularly when cues are subtle, such as those conveyed through timbre. This paper explores how dynamic stage lighting, driven by physiological signals, can enhance narrative comprehension in orchestral performance. Using the LightHearted interactive lighting system, di!erent characters of the Overture were mapped to distinct colored lights, whose intensities were dynamically modulated in real time by the heart rates of the conductor and selected musicians. This integration aimed to convey subtle narrative cues to the audience in real time. Audience feedback suggests that this approach not only clarifies musical narratives but also enhances the overall experience. Keywords: music performance ·multimodality ·user experience 1Introduction Music has long been a powerful medium for storytelling. In programmatic music works such as Berlioz’s Symphonie Fantastique or Tchaikovsky’s Romeo and Juliet,thenarrativeunfoldsthroughorchestraltexture,requiringaudiencesto interpret musical cues actively [12]. Program notes and pre-concert talks have traditionally aided audience comprehension of such music. However, pre-concert presentations may not always be available, and program notes require pre-concert reading and rely on language to shape the listener experience [7]. In some cases, increased awareness of narrative structures can detract from enjoyment [7], and may limit the audience’s interpretive autonomy [4]. Previous work highlights the tension between ine!able musical meaning and our need to articulate it with words. This paper explores one of our investigations into alternative approaches to conveying musical information: a performance system utilising an interactive multi-channel lighting system in a concert hall (Figure 1). We hypothesise that integrating music with light can deepen the audience’s understanding and enjoyment of the music, beyond simple auditory– visual pairing. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 447 2Christodoulouetal. Fig.1: The dynamic lighting system was based on displaying changing colours on the organ pipes above the orchestra. We developed a system in which the stage lights reflect the dramatic characters of Tchaikovsky’s overture Romeo and Juliet (TH 42, "W 39), using the following mapping: Instrumental Timbre ÑCharacter ÑAssociated Light Colour. Romeo is represented by two instruments—viola and cello—and is mapped to shades of blue light. Juliet is represented by flute and clarinet and is mapped to shades of red light. Whenever the “love theme” is played by both instruments that represent Romeo and Juliet, we use purple lighting. Throughout the rest of the piece, we display a yellow light. The development of the interactive lighting system, including technical details, has been presented elsewhere [2]. Here, we ask: RQ1 To what extent did the audience comprehend the connection between sound, light, and character information? RQ2 Does the system support the communication of the musical narrative? RQ3 Does lighting manipulation enhance the enjoyment and overall experience of the audience in an orchestra concert? This study aims to create a more enjoyable music experience by utilising the concert hall’s lighting system to “describe” dynamic and expressive elements of the performance. It also provides a non-linguistic, accessible channel for conveying narrative information, enhancing the experience for a diverse audience. Lastly, the data collected contributes to a dataset for analysing the influence of Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 448 Title Suppressed Due to Excessive Length 3 visual information on music performance perception, a topic of interest in many music research subdisciplines. 2Background 2.1 Program music Program music refers to instrumental compositions that aim to convey a narrative or communicate extra-musical ideas such as characters or stories. Listeners are invited to imagine these narratives through sound alone [8]. Program music was introduced through Western classical music through works such as Claude Debussy’s “L’Apres Midi D’un Faune” or Camille Saint-Saëns’ “Carnival of the Animals”. For listeners without formal music training, understanding the connection between sound and abstract storytelling can be challenging. 2.2 Music and lighting Research shows that combining sound and visuals significantly shapes cognitive and a!ective audience responses [5]. Although concert lighting traditionally sets the atmosphere, studies indicate that it strongly influences emotional responses to music [9]. Visual cues also impact auditory perception, as demonstrated by studies of the underlying mechanisms of music emotions [10]. We extend this by designing visual elements (light) in relation to the musical sound. Music and light have been paired from Wagnerian gesamtkunstwerk [11] and Xenakis’ Polytopes [6] to contemporary sound-art installations [3]. In program music, detecting recurring musical motifs (leitmotifs) associated with extramusical ideas is cognitively demanding [1]. Timbre plays a key role, as in Romeo and Juliet, where orchestration distinguishes motifs. 3TheAarhusSymphonyOrchestraExperiment This research was conducted in two phases: an initial live concert performance and a subsequent controlled video-based experiment designed to validate the findings and create a control group. 3.1 The Interactive Lighting System LightHearted is a modular array-based framework for mapping ECG signals to lighting designs in real time [2]. It supports both trigger-based and continuous mappings, with user-defined function chains enabling expressive and dynamic lighting behaviours. ECG signals are acquired over the Open Sound Control (OSC) protocol from a variety of supported devices, processed, and mapped to lighting parameters. Communication with the lighting system is also handled via OSC, with current support for the GrandMA3 console. LightHearted is implemented in Python, is open-source, and o!ers high flexibility and customisation. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 449 4Christodoulouetal. Lighting System Mapping Functions Theme Detection ECG Sensors Light Intensity Light Colour Music Score Fig.2: The LightHearted system for music-to-lighting mapping. The music score is used to map the characters to colours, whose intensity is controlled by live ECG signals. In this paper, we use LightHearted to combine two mapping strategies. On the one hand, we map the musical characters to colours (derived from a thematic analysis of the score), and on the other hand, we map the physiological signals to light intensity (derived from live ECG data) (Figure 2). LightHearted consists of four core components: data acquisition, data processing, music analysis, and lighting control. Data Acquisition. Wireless ECG sensors were placed on seven musicians. These included two violists (Viola I and II), who played the theme for Romeo, and one flutist, who played Juliet’s theme. To avoid making it too apparent to the audience which instruments were connected to the story, we also placed sensors on a French horn player, the timpanist, the concertmaster, and the conductor. Signals are transmitted over OSC to the LightHearted software environment. Data Processing. Acustomapplicationderivedreal-timebeats-per-minute(BPM) values from each signal and computed an ensemble average. This average BPM was mapped to stage-light intensity: higher collective heart rates produced brighter lights, highlighting moments of heightened musical or physical activity. Music Analysis. To e!ectively map the musical themes to the colour maps, we conducted a thematic analysis of the piece. First, we identified and marked the major thematic sections in the orchestral score: the Friar Laurence Theme (introduction), the Conflict Theme (representing the Montagues and Capulets), the Development section, and the Coda. We then focused in detail on the Love Theme, identifying its initial presentation and subsequent transformations throughout the piece. We noted which instruments carry the theme at key moments: primarily the viola and English horn for Romeo, and the flute and clarinet for Juliet. These associations allowed us to map specific characters to both musical timbres and corresponding colours: blue for Romeo, red for Juliet, and purple when their themes overlapped or were intertwined. The Love Theme recurs in Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 450 Title Suppressed Due to Excessive Length 5 both the Development and Coda, often varied in orchestration, which we traced closely to maintain consistent character representation. The remaining orchestral texture (outside of the Love Theme) was mapped to a constant yellow light, symbolising the broader narrative. Lighting Control. While intensity was modulated continuously via physiological data, a member of our team acted as a lighting operator, manually adjusting the colour using a DMX controller following our score notes. This human-in-the-loop approach ensured musically expressive colour changes, while the autonomously driven intensity layer added a responsive, embodied dimension to the lighting. 3.2 The Concert Experiment LightHearted was deployed during a live performance by the Aarhus Symphony Orchestra in May 2025. The Overture from Romeo and Juliet concluded the concert. This was part of a research concert that featured other experiments. At the end of the performance, audience members were asked to scan a QR code printed on their programs and complete a questionnaire. To test the e!ect of the programmatic context, only half of the audience (rows 1–19, odd-numbered seats) received program notes indicating the instruments that represented Romeo and Juliet, respectively. The other half of the audience (rows 1–19, even-numbered seats) received no narrative context, only the titles of the pieces. A control group (rows 20–26) received no program notes at all. It should be noted that the specific colour mappings (blue for Romeo, red for Juliet, purple for both, yellow for none) were not revealed to any group. However, all audience members were informed that select musicians and the conductor were wearing ECG sensors and that their heart signals would influence the intensity of the lighting in real time. The post-concert questionnaire was designed to assess how the audience perceived the integration of lighting with musical structure and character representation. It included multiple-choice, Likert-scale (1–7), and open-ended questions. Participants reported their section, row, and seat number, enabling us to correlate responses with program note exposure and viewing position. Regarding Romeo and Juliet, participants rated their familiarity with the piece, how well the lighting matched the music, how distracting or enjoyable the lighting was, and to what extent they felt that the lighting helped them understand the structure of the music. Audience members with available program notes for the piece were asked to identify which colour represented Romeo and Juliet from multiple choices (red, blue, purple, yellow), followed by confidence ratings for each selection. In contrast, audience members without this information had to answer which instrument represented Romeo and Juliet, also from a multiple-choice question, o!ering pictures as a hint. All audience members were also asked whether the lighting created a sense of space or depth and whether they would attend a concert with a similar lighting system in the future. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 451 6Christodoulouetal. 3.3 The Video Experiment The study was based on a single concert performance; therefore, no control group was used. Consequently, we designed a follow-up study involving a small group of participants for whom there was no systematic relationship between lighting and music. This allowed us to test whether our system e!ectively conveyed any information, or whether audience interpretations were instead influenced by other factors, such as pre-existing colour associations or bias. For one week, we experimented with video recordings of the live performance. We preserved the original audio and visuals of the musicians, but replaced the lighting colours with a randomly generated sequence. We edited the videos in such a way that the new lighting mimicked the timing and frequency of the original, but the colours were completely uncorrelated with the musical sound. We recruited 10 volunteers who, after viewing the video, were prompted to complete a digital questionnaire identical to the one used in the concert study. Data collection is ongoing. 4Results We collected N“355 questionnaire responses from the live concert and N“10 responses from the controlled experiment. The audience’s feedback was generally positive. Familiarity with the piece was low (6.6%), suggesting that responses were largely uninfluenced by prior knowledge. Most of the attendees found the lighting to be more enjoyable than distracting and expressed a strong interest in attending similar concerts in the future (Figure 3). While the lighting was seen to enhance spatial depth, its role in clarifying the musical structure was less clear, potentially influenced by the introduced bias. Our results also reveal distinct associations for the “Romeo” and “Juliet” themes, with interesting and sometimes counterintuitive patterns in audience confidence (Figure 4). In more detail, for Romeo’s theme, blue was the most commonly associated colour, followed by red. Confidence was also highest for these two colours. Regarding the instruments, the viola was the most frequently guessed instrument for Romeo’s theme, followed by the cello, which falls into the (a) Audience perception of lighting distraction, enjoyment, and future interest. (b) Audience perception of spatial depth and structural clarity enabled by lighting. Fig.3: Audience evaluations of the lighting system during the concert. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 452 Title Suppressed Due to Excessive Length 7 Fig.4: Audience guesses and confidence ratings for the musical themes representing Romeo and Juliet. same instrument category. For Juliet’s Theme, purple was the most frequently guessed colour, followed by red. The flute was the overwhelmingly chosen instrument associated with Juliet’s theme. However, average confidence was relatively low for this popular choice. The highest confidence was reported for trombone, an instrument that received almost no guesses. In the control group, opinions on the fit between lighting and music were mixed (avg. 3.5/7), as was perceived structural support (3.25/7). Nevertheless, the experience was generally positive: participants found the lighting enjoyable (4.13/7), only moderately distracting (3.63/7), and unanimously expressed interest in future concerts with similar lighting. Most were unfamiliar with the music, yet 75% identified red for Romeo and purple for Juliet. Despite this agreement, confidence in these selections was generally low. 5DiscussionandConclusions This study explored how the integration of lighting and music can support audience comprehension in programmatic music. Audience responses to the lighting system were generally positive, with many reporting enhanced enjoyment and a stronger sense of spatial depth, even among those unfamiliar with the piece. Feedback from the control group was more mixed: while participants found the lighting enjoyable and only moderately distracting, they expressed uncertainty about its connection to the music’s structure. Nonetheless, all indicated interest in attending future concerts featuring similar systems. Further investigation into the colour association answers is needed. Although the lighting did not improve narrative understanding, this outcome highlights the need for refined design and more controlled environments. Automated systems may o!er clearer mappings between musical and visual elements. Overall, our findings suggest that multimodal approaches can enrich audience engagement with orchestral music and warrant further exploration in innovative performance design. A systematic investigation into why specific visual associations emerge, such as the consistent selection of purple for Juliet and the variable choices for Romeo, could be a direction for future work. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 453 8Christodoulouetal. Data availability. All anonymised data are openly available at https://zenodo.org/ records/15865756 Ethics statement. This work was conducted following our institution’s ethical standards. All personal data is stored according to the GDPR. Disclosure of Interests. The authors have no competing interests in this paper. Acknowledgments. The Research Council of Norway supported this study through projects 262762 (RITMO), 324003 (AMBIENT), and 322364 (fourMs). References 1. Antovi", M., Stamenkovi", D., Figar, V.: Association of Meaning in Program MusicOn Denotation, Inherence, and Onomatopoeia. Music Perception 34(2), 243–248 (Dec 2016). https://doi.org/10.1525/mp.2016.34.2.243 2. von Armin, H.A., Christodoulou, A.M., Burnim, K., Upham, F., Kelkar, T., Jensenius, A.R.: LightHearted—A Framework for Mapping ECG Signals to Light Parameters in Performing Arts. OSF (2025). https://doi.org/10.17605/OSF.IO/9SCDX 3. Basanta, A.: Compositional Strategies in Light and Sound Installations. Master’s thesis, Concordia University (Mar 2013), https://spectrum.library.concordia. ca/id/eprint/976949/ 4. Bennett, D., Ginsborg, J.: Audience reactions to the program notes of unfamiliar music. Psychology of Music 46(4), 588–605 (Jul 2018). https://doi.org/10.1177/0305735617721339, publisher: SAGE Publications Ltd 5. Geringer, J.M., Cassidy, J.W., Byo, J.L.: Nonmusic Majors’ Cognitive and A!ective Responses to Performance and Programmatic Music Videos. Journal of Research in Music Education 45(2), 221–233 (Jul 1997). https://doi.org/10.2307/3345582, publisher: SAGE Publications Inc 6. Harley, M.A.: Music of Sound and Light: Xenakis’s Polytopes. Leonardo 31(1), 55–65 (1998), https://muse.jhu.edu/pub/6/article/607587 7. Hellmuth Margulis, E.: When program notes don’t help: Music descriptions and enjoyment. Psychology of Music 38(3), 285–302 (Jul 2010). https://doi.org/10.1177/0305735609351921, publisher: SAGE Publications Ltd 8. Hepokoski, J.: Program music. In: Aesthetics of Music. Routledge (2014), num Pages: 22 9. Hsiao, S.W., Chen, S.K., Lee, C.H.: Methodology for stage lighting control based on music emotions. Information Sciences 412-413,14–35(Oct2017). https://doi.org/10.1016/j.ins.2017.05.026 10. Juslin, P.N., Västfjäll, D.: Emotional responses to music: The need to consider underlying mechanisms. Behavioral and Brain Sciences 31(5), 559–575 (Oct 2008). https://doi.org/10.1017/S0140525X08005293 11. Silverthorne, D.V.: Music, Art and Performance from Liszt to Riot Grrrl: The Musicalization of Art. Bloomsbury Publishing USA (Oct 2018), google-Books-ID: YKtqDwAAQBAJ 12. Thompson, W.F., Bullot, N.J., Margulis, E.H.: The psychological basis of music appreciation: Structure, self, source. Psychological Review 130(1), 260–284 (2023). https://doi.org/10.1037/rev0000364, place: US Publisher: American Psychological Association Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 454