Integrating Virtual Reality into Music Performance for Concert Audiences

Integrating Virtual Reality into Music Performance for Concert Audiences Elois Zelada1and Anıl Çamcı2 University of Michigan 1[email protected] 2[email protected] Abstract. An increasing number of practitioners are exploring the affordances of Virtual Reality (VR) in music performance. However, sharing the VR performer’s experience with live audiences remains a challenge. In this paper, we present Bardo, a work for flute, fixed electronics, and VR, along with the creative practice developed in collaboration with a composer and flutist. We designed an audio-reactive VR environment inspired by the themes of the composition and features of the flute performance. The environment incorporates Fifth Wall, a custom camera system that implements cinematographic techniques, which can be controlled by the user, stochastic processes, or a real-time music information retrieval engine. This system provides traditional concert audiences with a cinematic view of the VR environment wherein the performer is situated. The work presented in this paper serves as a case study on bridging the experiential gap between VR performers and traditional audiences and identifies new opportunities for expressive and accessible VR music performances. Keywords: Virtual Reality ·Music Performance ·Audiovisual Music 1Introduction Virtual Reality (VR) offers unique possibilities for music performance through audiovisual experiences that defy the rules of physics and display magical qualities. It also supports the creation of purpose-built virtual environments that not only reflect the themes of a performance but often serve as musical interfaces themselves. As a result, VR is being used to develop practices where a performer interacts with virtual objects, agents, and environments while situated on stage in front of a live audience. A core challenge in these performances is conveying the performer’s virtual experience to the audience. While consumer adoption of VR technology has grown in recent years, it remains unrealistic to expect concert audiences to bring or be provided with dedicated VR systems. Supplying headsets at scale is costly, technically demanding, and difficult to maintain. Consequently, most VR performances rely on screens or projections to present the virtual world to the audience. However, effectively displaying these immersive experiences to live audiences continues to be challenging [11]. Traditional methods, such as projecting firstperson perspectives from the performer’s headset, can suffer from limited and Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 838 2E.ZeladaandA.Çamcı unstable views [13]. Alternatively, multiple virtual cameras can provide a more comprehensive perspective but require careful setup and management [2]. In response to these challenges, this paper examines Bardo,apieceforflute, electronics, and VR, as a case study on the integration of VR technology in music performance. The project entailed a close collaboration with composer Erik Santos and flutist Amy Porter to develop a novel creative practice. Central to this effort was the incorporation of Fifth Wall, a custom camera system that we designed to deliver engaging and cinematic representations of VR performances. This collaboration resulted in a showcase that demonstrated the feasibility and expressive potential of VR for music performance while exploring the challenges and opportunities in showcasing virtual environments to concert audiences. 2RelatedWork Extended Reality (XR) technologies are increasingly used for designing custom virtual systems for music performance. These works offer novel ways to create immersive and interactive settings that enhance the performer’s ability to express musical ideas [9,6,5,15]. In parallel, previous work has focused on developing methods for presenting these performances to audiences [4,3,13,14]. One example is Santini’s use of an AR head-mounted display in piano performance, using AR as both an interface and an audience visualization tool, overlaying virtual elements on the piano and projecting the performer’s augmented view onto a display for the audience [10]. In another example, where XR is used for visualizing a live music performance, Weinel’s Cyberdreams project utilizes VR to bring audiences into fictional environments governed by the music. Rather than relying on traditional audio-reactive visuals, Weinel argues for XR experiences that evoke symbolic meanings to enhance the music’s imaginative qualities through synaesthetic and immersive 3D spaces [13]. Despite recent advances in VR technology, sharing immersive virtual environments with large audiences remains limited by practical challenges. Previous research has highlighted the disparity between the VR performer’s experience and the audience’s often passive observation; accessibility, cost, and logistical constraints in providing VR devices to audiences mean that most viewers experience these performances through 2D displays [11, 13, 12]. Two common methods for displaying VR content to audiences are: (1) displaying the performer’s first-person headset view, and (2) using third-person virtual cameras placed in virtual space. The first-person view is easy to implement but has notable drawbacks. VR headsets have a limited field of view, wherein the effects of subtle head movements become amplified when this view is projected on a large screen; this results in a narrow and shaky perspective that can be disorienting for viewers. This can be addressed by smoothing and widening the field of view captured by a virtual camera attached to the headset, as demonstrated in [1]. However, this approach still ties the audience’s perspecProc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 839 Integrating VR into Music Performance 3 tive to that of the performer, limiting the spatial context and placing the burden of visual curation on the performer. The second approach (i.e., a third-person virtual camera setup) offers greater flexibility and a more extensive view of the virtual environment. However, this requires pre-planned camera placement or live control, which can be technically demanding and may constrain the performer’s movement or freedom to improvise, especially in room-scale setups that overlap virtual and physical spaces. A project that explores this approach is Hamilton’s VR string quartet, where audiences are shown the virtual space shared by co-located performers. Hamilton adopts a method similar to eSports broadcasts with a set of virtual cameras projecting 2D views of each performer [7]. Despite its practicality, displaying virtual content on 2D screens can diminish the depth and spatial scale fundamental to XR experiences. In this context, Berthaut et al. emphasize the importance of scenography in VR performances. Their framework for evaluating Immersive Virtual Musical Instruments underscores the need to balance the performer’s perspective with clear and compelling visuals for the audience in order to ensure that the virtual environment conveys a coherent narrative and enhances the audience’s understanding of and emotional connection with the performance [2]. To that end, Berthaut et al. propose several dimensions for evaluating the scenography of Immersive Virtual Musical Instruments, three of which are particularly relevant to our project: “audience immersion” describes the degree to which the audience perceives the VR performance and it can be strengthened by display techniques; “gestures continuity” relates to how well the musician’s physical gestures are represented in the virtual domain; and “from virtual to physical” indicates the degree of integration between virtual and physical environments [2]. Additionally, prior literature on 3D viewpoint control in VR highlights techniques such as camera paths and point-of-interest selection that optimize audience comprehension and immersion [8]. 3 The Fifth Wall System Building on similar principles, our project explores strategies for delivering VR music performances to concert audiences. To address the limitations of current display methods, we developed Fifth Wall, a virtual cinematography system that implements dynamic camera behaviors that can be controlled by the VR performer’s behaviors, audio features extracted from the music, external input, or stochastic processes. The system manages an array of virtual cameras, each operating under distinct algorithms that emulate cinematic filming techniques. Developed for the Unity game engine, Fifth Wall includes several core manager scripts that coordinate the system’s adaptive camera behavior as shown in Fig. 1. Camera Algorithm Manager enables the live switching between algorithms that implement camera movement techniques, such as trucking, dollying, panning, and zooming, as well as first-person and third-person views. Furthermore, it allows the control of global camera parameters such as movement speed Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 840 4E.ZeladaandA.Çamcı Fig.1: Inspector view of the Fifth Wall prefab, the main user interface for controlling the system in Unity. The modular organization includes components for managing camera algorithms and capture, interest, and focus objects. The diagram below illustrates the control flow between system components and users. and placement distance. The Camera Algorithm Manager can also be controlled externally via Open Sound Control (OSC). To this end, we designed a music information retrieval (MIR) engine using IRCAM’s Max Sound Box.1This allows features of the Fifth Wall system, such as camera weight, selection, and movement speed, to be controlled automatically by applying common MIR techniques, such as pitch, volume, onset, and spectral centroid detection, to the music in real time. The output of the MIR engine can be mapped to the Camera Algorithm Manager parameters over OSC. Capture Objects Manager detects which game objects the performer is gazing at or close to in the virtual environment. The duration of their gaze or their proximity affects the weight assigned to an object. Interest Objects Manager maintains a dynamically weighted list of game objects that the camera system could focus on. This list can be populated either automatically by the Capture Objects Manager or manually by adding game objects to the list and assigning them a weight. Focus Object Manager selects an interest object for the camera system to focus on based on their weight. The focus object can be manually overridden to showcase a different object at any given time. Game objects can be given a fixed weight so that they are not affected by the Capture Objects Manager. 1https://forum.ircam.fr/projects/detail/max-sound-box/ Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 841 Integrating VR into Music Performance 5 For instance, the performer avatar can be given a high fixed weight so that it is showcased more frequently by the camera system. 4Bardo As a case study on the integration of VR technology in music performance, we collaborated with composer Erik Santos and flutist Amy Porter on Bardo, a piece for flute, electronics, and VR. Rather than working with a previously composed piece, we conceived composition, performance, system development, and VR design as intertwined aspects of our collaboration. Guided by the goals of our system research and Porter’s narrative and thematic prompts, Santos composed the music for flute and electronics while we developed the virtual environment wherein Porter would perform the piece. Over the course of two years, we engaged in a multimedia performance practice to develop and evaluate the Fifth Wall system. A short demonstration of this work can be viewed at this link.2 4.1 Environment Design Inspired by the Tibetan concept of a transitional state, Bardo employs both musical and VR elements to evoke liminality. By drawing on the landscapes of Whitefish Point, Michigan, the VR environment juxtaposes real-life landmarks, like the lighthouse by the lake and surrounding town and forest, with surreal elements, such as a ribcage modeled after the flutist’s x-rays and audio-reactive particle systems, as shown in Fig. 2a. The interplay between the real and the imaginary is aimed at amplifying the themes of the music while bridging the physical and virtual aspects of the performance. The VR environment situates the performer on a dock, which is mapped to the physical performance space in room scale. While this limits the performer’s navigation of virtual space for safety purposes, the virtual environment expands much further beyond this area, encompassing landmarks from our reference location. 4.2 Audiovisual Mappings In Bardo,soundsoftheliveperformanceandthevisualfeaturesoftheVRenvironment are closely linked. The audio input from the flute and fixed electronics are analyzed in real time using our MIR system. The retrieved information is sent to Unity over OSC and is mapped to various features of the Fifth Wall system, such as camera movement and transitions, as well as environmental features of the VR space. For instance, we incorporated various particle systems in the VR environment to visualize the flute’s amplitude as shown in Figs. 2c and 2d. The visual elements also respond to musical changes in the fixed electronics. For instance, loud transients trigger lightning effects that alter the scene’s 2https://myumi.ch/g3bWP Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 842 6E.ZeladaandA.Çamcı (a) A wide shot of the VR environment modeled after Whitefish Point. (b) Lightning triggered by a loud transient in the audio track. (c) The ribcage surrounded by audioreactive particles. (d) Avatar on the dock with particles that react to the flute’s sound in real time. Fig.2: Screenshots of the VR environment with audio-reactive visual elements. brightness, as shown in Fig. 2b. Other audio features influence post-processing effects: chromatic aberration reacts to perceived sharpness, distorting the image during harsher moments, while changes in the spectral centroid subtly adjust the scene’s color palette to reflect harmonic shifts. Noisiness in the fixed electronics is mapped to a rain particle system and cloud density, dynamically changing the weather with the music. These subtler elements, though not always immediately noticeable, contribute to the dynamism of the environment. These design choices are intended to create a responsive space that immerses the performer in an interactive VR environment while simultaneously generating a cinematic, real-time visualization for the audience. Through these mappings, we strove to make the VR environment more expressive for the performer and more engaging for the audience. 4.3 Performer Avatar The VR component of Bardo enables the performer’s actions to be reflected in a digital avatar using Meta’s Movement SDK, as shown in Fig. 3. During the performance, the Meta Quest Pro headset equipped by the performer captures joint-movement data, which is then mapped to a custom avatar created using the Ready Player Me platform.3By mirroring the movements of the performer 3https://readyplayer.me Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 843 Integrating VR into Music Performance 7 Fig.3: Flutist equipped with a VR headset in front of the projection screen, which displays the output of the Fifth Wall. on stage, the avatar helps anchor the audience’s view of the virtual environment in physical reality. 4.4 VR Performance As a culmination of these efforts, Bardo was performed at the Michigan Chamber Players Concert in Britton Recital Hall on April 7, 2024. The stage where Porter performed the piece was mapped for room-scale VR ahead of time. While Porter was performing the piece, the output of the Fifth Wall system was projected onto a large screen on stage, allowing the audience to experience the VR environment. The VR system was controlled both automatically based on live audio features and manually based on operator input. In this performance, the curation and weighting of interest objects, such as the lighthouse, dock, townhouses, forest, ribcage, and particle systems, were controlled by us as external operators. Our approach to this curation was informed by a predetermined narrative based on the composition’s structure. This curation was further refined through conversations with the artists to better support the creative intents and to reveal different aspects of the virtual environment over the course of the performance. For instance, at the beginning of the piece, where the musical elements are gradually introduced, we emphasized the forest and horizon while the sun was rising in the virtual environment. As the music progressed, we increased the weights of other virtual objects such as the performer’s avatar and the dock, while reducing that of the forest, prompting the camera algorithms to highlight these new focal points. This process continued throughout the piece, with weights dynamically adjusted to reveal different objects, while the camera changes and movements were controlled both automatically through musical features and manually by operator input. For manual changes in weights and camera selection alongside those triggered by the MIR engine, we used a MIDI device to control Fifth Wall over OSC through Max. This setup, for instance, allowed us to map a button to trigger a change in the camera type and use a knob to control the global speed of camera movement. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 844 8E.ZeladaandA.Çamcı In preparation for the performance, we conducted several rehearsals with Santos and Porter to collaboratively decide on the narrative unfolding of the VR space displayed to the audience. Furthermore, to provide the performer with a comfortable experience, we took active measures, such as setting a room-scale play area that they feel comfortable moving within and maintaining a high frame rate in the headset to avoid simulation sickness. 5Conclusion This project illustrates a novel integration of VR into live music performance through a close collaboration among artists and developers, resulting in a hybrid practice that expands both technological and artistic possibilities. Our work aims to address a growing need in VR music practices to make immersive content accessible and engaging for live audiences without relying on head-mounted displays. It explores the multimodal and interactive affordances of VR to enhance the expressive potential of a music performance, while dealing with key limitations of traditional VR display methods, such as the instability of first-person views and the rigidity of fixed virtual camera setups. Through this project, we explored the design of a custom-built virtual environment that visually reflects the symbolic elements of a music composition. To that end, we used audiovisual mappings to integrate sound and virtual objects. The use of the Fifth Wall system enabled responsive visual storytelling through the real-time interaction of musical features with virtual objects. Through a combination of manual and algorithmic control, the system generated a curated, cinematic perspective of the virtual environment. The implementation of the performer avatar in Bardo was intended to bridge the perceptual gap between the physical and virtual components of the performance and help both the flutist and the audience maintain a sense of embodiment and spatial continuity across realities. Although we have been actively engaged with a performer throughout the development process to improve not only the system but also the performer’s experience in VR, we aim to evaluate Fifth Wall with other performers to understand its broader applicability to music performance. We also intend to carry out a survey of the audience experience to identify areas for improvement in the camera system. We hope that this project can serve not only as a model for addressing current technical challenges in music performance using XR technology but also as an approach to developing a creative practice with such technology through continued collaboration among all stakeholders, including artists, performers, and system designers. Acknowledgments. This project was supported by an Arts Initiative Exploration Pilot Grant a the University of Michigan. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 845 Integrating VR into Music Performance 9 References 1. Atherton, J., Wang, G.: Curating Perspectives: Incorporating Virtual Reality into Laptop Orchestra Performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 154–159. Birmingham, United Kingdom (2020). https://doi.org/10.5281/zenodo.4813290 2. Berthaut, F., Zappi, V., Mazzanti, D.: Scenography of immersive virtual musical instruments. In: Proceedings of the IEEE VR Workshop on Sonic Interaction in Virtual Environments (SIVE). pp. 19–24. IEEE, Minneapolis, USA (2014). https://doi.org/10.1109/SIVE.2014.7006285 3. Enriquez, K., Palacios, M., Pallo, D., Guerrero, G.: SENSE: Sensory component VR application for hearing impaired people to enhance the music experience. In: 2020 15th Iberian Conference on Information Systems and Technologies (CISTI). pp. 1–6. IEEE, Sevilla, Spain (2020). https://doi.org/10.23919/CISTI49556.2020.9140447 4. Fernandes, R., Gaonkar, A.P., Shenoy, P.J., Rodrigues, A.P., A., M.B., Padmanabha, V.: Efficient Virtual Reality-Based Platform for Virtual Concerts. In: Tyagi, A.K. (ed.) Multimedia and Sensory Input for Augmented, Mixed, and Virtual Reality, pp. 148–164. IGI Global, USA (2021). https://doi.org/10.4018/978-1-79984703-8.ch008 5. Frame, C.: Concerts of the Future: Designing an interactive musical experience in VR. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 359–366. Utrecht, Netherlands (2024). https://doi.org/10.5281/zenodo.13904880 6. Graham, R., Bridges, B., Manzione, C., Brent, W.: Exploring Pitch and Timbre through 3d Spaces: Embodied models in virtual reality as a basis for performance systems design. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 157–162. Copenhagen, Denmark (2017). https://doi.org/10.5281/zenodo.1176207 7. Hamilton, R.: Trois Machins de la Grâce Aimante: a Virtual Reality String Quartet. In: Proceedings of the 2019 International Computer Music Conference. New York, USA (2019) 8. Jankowski, J., Hachet, M.: Advances in Interaction with 3D Environments. Computer Graphics Forum 34(1), 152–190 (2015). https://doi.org/10.1111/cgf.12466 9. Masu, R., Bala, P., Ahmad, M., Correia, N.N., Nisi, V., Nunes, N., Romão, T.: VR Open Scores: Scores as inspiration for VR scenarios. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 109–114. Birmingham, United Kingdom (2021). https://doi.org/10.5281/zenodo.4813262 10. Santini, G.: Augmented Piano in Augmented Reality. In: Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 411–415. Birmingham, UK (2021). https://doi.org/10.5281/zenodo.4813449 11. Turchet, L., Hamilton, R., Çamcı, A.: Music in Extended Realities. IEEE Access 9,15810–15832(2021).https://doi.org/10.1109/ACCESS.2021.3052931 12. Uro, P., Berthaut, F., Pietrzak, T., Wanderley, M.M., Grisoni, L.: Strategies for the Reconciliation of Artistic Intent and Technical Constraints in Mixed Reality Performances. In: Proceedings of the 2025 ACM Designing Interactive Systems Conference. pp. 1603–1614. Funchal, Portugal (2025). https://doi.org/10.1145/3715336.3735738 13. Weinel, J.: Cyberdreams: Visualizing music in extended reality. In: Earnshaw, R., Liggett, S., Excell, P., Thalmann, D. (eds.) Technology, Design and the Arts - OpProc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 846