Singing Without a Larynx: A Vowel-Based Real-Time Interface for Embodied Vocal Performance

All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Singing Without a Larynx: A Vowel-Based Real-Time Interface for Embodied Vocal Performance Gala Lucia Gonzalez Barrios1 [0009-0005-1213-1168] and Ico Bukvic2[0000-0001-5129-3921] 1 Virginia Tech, Virginia, 24060, USA 2 Indiana University, Indiana, 47405, USA [email protected], [email protected] Abstract This demonstration introduces Singing Without a Larynx, an early prototype of a vowel-based real-time vocal interface designed to support creative expression for individuals who have undergone total laryngectomy. Implemented in JavaScript with MediaPipe’s facial landmark detection, the system identifies mouth shapes corresponding to five cardinal vowels /a, e, i, o, u/ and humming (/mː/ nasal phonation) and maps them to sound synthesis in WebPdL2Ork. The project addresses identity restoration through creative expression. Through vowel gestures, openness intensity, and transitions, users shape timbres that range from resonant /A/ tones to crystalline /I/ textures. As a proof of concept, this prototype integrates perspectives from HCI, speech science, vocology, and STS. The demo showcases vowel classification and sonic output, inviting reflection on aural diversity and embodied voice-making to go beyond techno-ableism. Keywords: Assistive Voice Technology, Human Centred Design, Aural Diversity, Sonic Interaction Design, Embodied Vocal Performance 1 System Description This demonstration introduces Singing Without a Larynx, an early-stage, vowel-based real-time vocal interface designed to support creative expression for individuals who have undergone total laryngectomy. Built in JavaScript with MediaPipe’s facial landmark tracking [1], the system detects lip-region movements. Vowel classification relies on normalized ratios of lip height to width and distances between upper and lower lip points, mapped to five vowels /a, e, i, o, u/ and humming (/mː/ nasal phonation).Transitional gestures, such as gradual widening of the mouth, are interpreted as continuous controls, enabling expressive glides rather than only discrete categories. This approach remains preliminary, serving as proof-of-concept rather than a validated classifier. Figure 1 illustrates the complete system architecture and data flow. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1027 2 G. Gonzalez and I. Bukvic The system operates entirely in web browser, requiring only a webcam. MediaPipe’s 468-point facial mesh provides real-time tracking at over 30 frames per second, with a specific focus on the lip region landmarks. The classifier calculates normalized ratios between key points: outer lip width (landmarks 61–291), inner mouth height (landmarks 13–14), and lip thickness variations. These geometric features are then mapped to vowel categories using threshold-based classification. In addition to facial tracking, the interface integrates the mobile phone as a gestural controller. This design choice intentionally redistributes interaction from the upper body (torso, shoulders, neck, head) to the hand, offloading physical strain and enabling a broader range of fluid, embodied, and continuous gestures. By leveraging hand, eye coordination to shape and modulate sound output, the system aims to foster a more intuitive, flexible, and expressive interaction. We hypothesize that this shift significantly expands the expressive potential of the interface and contributes to a more natural interaction experience, a claim to be evaluated through future user studies. Fig. 1. System architecture showing the browser-based real-time processing pipeline from dual inputs (webcam and mobile phone) to audio output. The system maps six vowel articulations (/a, e, i, o, u/ and humming) to distinct timbral characteristics through sampling and crossfading, with gestural controls for pitch, amplitude, and brightness modulation. Sound synthesis is implemented in WebPdL2Ork [2] through sampling with cross fading and resampling (for pitch adjustment). Each vowel triggers formant-inspired filters and harmonic layers: /A/ emphasizes lower resonances for warmth, while /I/ highlights upper partials for crystalline textures. Mouth opening dynamically modulates brightness, providing phrasing and timbral variation. Closeness to camera modulates amplitude and up-down movements modulate pitch. These mappings are exploratory and designed to test feasibility, not to replicate natural vocal quality. Previous systems such as VocalTractLab [3], Vocaloid [4], and vowel-mapping controllers [5][6] have explored similar terrain but often target expert performers. Singing Without a Larynx differs by prioritizing participatory design and identity restitution for laryngectomy patients. With approximately 12,500 new cases annually in the United States, this community, predominantly working-class individuals exposed Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1028 Singing Without a Larynx: A Vowel-Based Real-Time Interface 3 to occupational carcinogens, remains systematically under-researched and underresourced [7][8][9]. By reframing the voice interface as a site of creative empowerment, the demonstration contributes to ongoing discussions on inclusive sonic technologies and the cultural meanings of voice [10]. 2 Demonstration Activities The demonstration will follow a structured 60-minute format. It will begin with a 5minute introduction situating the demographics of the laryngectomee community, highlighting its status as an under-represented population, lacking advocacy structures, and marginalized after surgery [7][8][9]. This will be followed by a 15-minute guided tour of the prototype web app, showing vowel detection and sound mappings in real time, with live demonstration of the five vowel classifications and their corresponding sonic outputs. Attendees will then engage in a 25-minute rapid prototyping exercise: using provided sheets and pencils, participants will sketch potential functionalities, user flows, or alternative mappings they imagine for the system. This hands-on activity draws on human-centred design methods that emphasize quick, low-fidelity prototyping to stimulate creativity and dialogue [11]. The session will conclude with a 15-minute group discussion reflecting on participatory approaches to designing sonic interfaces and gathering feedback on the prototype's potential for community-based development. 3 Demonstration Purpose The purpose of this demonstration is to present Singing Without a Larynx as an early prototype that illustrates the conceptual and technical foundations of a broader research agenda on voice and identity restitution after total laryngectomy, fostering reflection on how voice interfaces can develop through interdisciplinary collaboration to navigate questions of identity. By combining hands-on interaction with the prototype, a rapid prototyping exercise, and group discussion, the demo aims to engage the CMMR community in considering how creative expression and embodied interaction can expand the role of assistive technologies beyond techno-ableism and technosolutionism [12][13]. The ultimate goal is to gather constructive feedback for next iterations of the development while situating the voice as both a technical and cultural phenomenon. Following this demonstration and IRB approval, the next phase will involve participatory design sessions with laryngectomees in both the United States and Argentina, incorporating their lived experiences and creative agency into the system's evolution. Acknowledgments. This project is supported by Virginia Tech’s Human-Centered Design Interdisciplinary PhD Program. The author gratefully acknowledges the guidance of her doctoral committee: Dr. Ashley Shew, Dr. Ivica Ico Bukvic, Dr. Vrushali Angadi, Dr. Myounghoon Jeon, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1029 4 G. Gonzalez and I. Bukvic Dr. Cecilia Rikap and Dr. Fabian Preto-Nanez. Special thanks to Dr. Bukvic for co-developing the system, and to collaborators across speech-language pathology, vocology, HCI, HFE, and STS for their interdisciplinary contributions. Disclosure of Interests. The author declares no competing interests relevant to the content of this demonstration. References [1] Y. Kartynnik et al., "Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs," in Proc. CVPR Workshop on Computer Vision for Augmented and Virtual Reality, 2019. [2] I. Bukvic, “Pd-L2Ork: Extensible environment for laptop orchestra and classroom use,” in Proc. ICMC, 2011. [3] P. Birkholz, “Modeling Consonant–Vowel Coarticulation for Articulatory Speech Synthesis,” PLoS ONE, vol. 8, no. 4, e60603, 2013. [4] H. Kenmochi and H. Ohshita, “VOCALOID – Commercial singing synthesizer based on sample concatenation,” in Proc. INTERSPEECH, pp. 4011–4012, 2007. [5] J. Armitage, “Singing with your Hands: Exploring Vowel Mapping in Vocal Gesture Interfaces,” in Proc. NIME, 2015. [6] T. Park and S. Lee, “Embodied Control of Vocal Synthesis through Vowel Gesture Mapping,” in Proc. ICMC/NIME Joint Conf., 2019. [7] I. C. Cnossen et al., “A participatory design approach to develop a web-based self-care program supporting early rehabilitation among patients after total laryngectomy,” Folia Phoniatr. Logop., vol. 67, no. 4, pp. 193–201, 2016. [8] I. Brook, “Social isolation and loneliness in laryngectomees,” Eur. Ann. Otorhinolaryngol. Head Neck Dis., Aug. 2024, doi: 10.1016/j.anorl.2024.07.006. [9] J. M. Costa, M. López, J. García, X. León, and M. Quer, “Impact of Total Laryngectomy on Return to Work,” Acta Otorrinolaringol. Engl. Ed., vol. 69, no. 2, pp. 74–79, Mar. 2018, doi: 10.1016/j.otoeng.2017.02.013. [10] M. Dolar, A voice and nothing more. In: Short Circuits. Cambridge, MA: MIT Press, 2006. [11] IDEO.org, The Field Guide to Human-Centered Design. IDEO, 2015. [12] A. Shew, Against Technoableism: Rethinking Who Needs Improvement, in Catalyst: Feminism, Theory, Technoscience, vol. 5, no. 1, 2019. [Online]. Available: https://catalystjournal.org [13] E. Morozov, To Save Everything, Click Here: The Folly of Technological Solutionism, New York: PublicAffairs, 2013. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 1030