Belly of the Beast

This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia © 2025 Copyright held by the owner/author(s). Belly of the Beast JACOB HEDGES, University of Technology Sydney, Australia DANIEL GARRETT, Sydney, Australia JAXON SHARP, Sydney, Australia Program Notes Belly of the Beast is an interactive VR music experience that explores how new technology has the potential to challenge the role of the audience in popular music. By leveraging hand tracking, spatial audio, and unique virtual instruments, this work reimagines how we interact with music as the listener. Participants dynamically manipulate a musical composition in 3D space, engaging as producer, performer, and audience simultaneously. Project Description With the continued advancement of XR technologies, unique possibilities for detailed interaction are emerging, opening new pathways for musical expression for both performers and audiences. This project seeks to explore how utilizing a number of these interactions can allow for a shift in the way we experience music, presenting a spatial composition in which the participant plays an active role. Built using Unity and Wwise for the Meta Quest 3, Belly of the Beast is a 15-minute interactive musical experience that immerses users in a virtual world where they manipulate the soundscape and composition by interacting with virtual objects and interfaces. During development, the decision was made to forego controllers and instead rely on the Meta Quest 3's hand-tracking capabilities for more natural interactions. This decision not only eliminates the need for users to learn complex controls but also informs the design of the virtual interfaces, ensuring they work effectively with simple hand gestures. Fig. 1. A view from inside the VR demo showcasing the hand tracking interaction with a virtual instrument interface that triggers a granular synthesizer. Once the user places on the VR headset, they are invited to explore the virtual space around them and interact with what they see and hear. Audio stems of the composition are spatialized around the listener as floating orbs using binaural processing, with their positions adjustable via hand tracking NIME ’25, June 24–27, 2025, Canberra, Australia Hedges et al. interactions. A virtual keyboard-like interface, tuned to the song’s key, invites the user to contribute their own melodies via direct hand interaction. Additionally, a granular synthesizer, featuring a bespoke 3D interface, allows users to manipulate parameters of the synthesizer with creative freedom (Figure 1). Furthermore, a grid interface is introduced that explores a common game music technique known as vertical resequencing [1], in which the stems of the composition are dynamically remixed based on game events. The grid interface opens this mechanic up to the user, empowering them to rearrange the composition by selecting or muting individual instrumental parts, all synchronized to the song’s tempo and meter. This hands-on approach invites exploration and creativity, giving users the ability to deconstruct and rebuild the piece in real time. Throughout the virtual environment, audiovisual reactive elements subtly enhance the experience. A 3D model of a face animated using motion capture sings along with the composition, adding a sense of social presence - where virtual beings feel like they are ‘really there’, heightening the immersive experience [2]. Particles and rays of light move and change in response to the music, creating a visual expression of the sound in the virtual space. A giant human figure looms over the scene, its presence culminating in the dramatic act of swallowing the user at the demo’s conclusion, creating a visceral moment of narrative closure. This work embodies the spirit of entangled musical interfaces, where gestures, sound, and visuals converge to redefine the boundaries of musical performance in virtual reality. By engaging with the interplay of technology, creativity, and user agency, the experience highlights the transformative potential of VR as a medium for new music-making practices. The video link provided offers a view at an early prototype demo of the experience presented during the XR showcase at SXSW Sydney 2024. The intended submission for NIME 2025 includes an updated and refined version of the experience containing new features and improved interactions. Performance notes The total duration of the experience is between 15 minutes per participant, depending on certain nonlinear events triggered by the participant. Each participant will be guided on the basic operation of the experience prior to starting and then continue until the end of the demo. Practical Requirements • One table • 4m x 4m space for VR room-scale exploration • Indoor venue with sufficient lighting for hand-tracking functionality • Sufficient power to run PC Technical Requirements • Meta Quest 3 VR headset • Windows PC and monitor to run the experience software • All necessary cables (checked and tagged) Performers to Bring • Meta Quest 3 VR headset Belly of the Beast NIME ’25, June 24–27, 2025, Canberra, Australia • PC and monitor • All cables • Posters and lighting for aesthetics Media Link(s) Video supplied in supplementary material. Acknowledgments This research was supported and funded by Dolby Australia and the Australian Government Research Training Program (RTP) Scholarship. We would also like to acknowledge the hard work of the production team: Jacob Hedges (Director/Audio lead/Composer), Daniel Garrett (Creative programming lead), Jaxon Sharp (Vocalist / Composer), and Brandon Doray (Additional animation). Ethical Standards To our knowledge, there are no ethical conflicts involved in the making and presentation of this work. References [1] Sweet, M. (2014). Writing Interactive Music for Video Games: A Composer’s Guide. Addison-Wesley Professional. [2] Lee, K. M. (2004). Presence, explicated. Communication theory, 14(1), 27-50.