Cybernesis: An Improvisational Performance System Exploring Gesture Control of Hardware Synthesizers

Cybernesis: An Improvisational Performance System Exploring Gesture Control of Hardware Synthesizers David Piazza CIRMMT Université de Montréal 200, av. Vincent-d’Indy, Montréal [email protected] Abstract Cybernesis is a performance piece that explores the interaction between human gesture, machine learning, and real-time sound synthesis. Using a Leap Motion controller, the performer’s hand movements are captured and analyzed by custom software. This gestural data trains a multilayer perceptron (MLP), a form of neural network, which in turn predicts and influences the internal state of a complex hardware sound synthesis system. The core of the performance lies in the real-time exploration of nonlinear mapping functions through linear regression, navigated spatially through the performer’s listening and intuitive hand movements. This creates a dynamic feedback loop where the performer and the model co-create the sonic output, positioning the learning algorithm not merely as a tool, but as an active participant in the improvisational process mediated by the performer’s embodied interaction. 1 Introduction: Machine Learning, Improvisation, and Embodied Interaction The intersection of machine learning (ML) and musical creativity continues to yield novel forms of expression and interaction. Cybernesis is presented as a contribution to this field, specifically focusing on live, improvisational performance using ML as a mediating layer between performer gesture and complex sound generation. The work leverages ML techniques to create intricate, evolving relationships between embodied human input and the parameters of a hardware-based sound synthesis system, Destiny+’s Model Q2 phase modulation synthesizer and Entanglement Space effects unit. Central to Cybernesis is the exploration of nonlinear mapping strategies, moving beyond direct one-to-one correspondences between gesture and sound (Hunt et al., 2002). Instead, a multilayer perceptron (MLP) is trained based on the performer’s hand movements detected by a Leapmotion infrared camera, using the Fluid Corpus Manipulation toolkit (Tremblay et al., 2021). This MLP learns associations between hand positions and sets of varying internal states within the sound synthesis hardware. The performer navigates this complex control space through listening and subtle hand movements, engaging in an improvisational dialogue with the system. During the piece, the performer is actively shaping the behaviour of the prediction mechanism and responding to the sonic results in a tight feedback cycle. The model acts as an intelligent intermediary, translating high-dimensional gestural input vectors into control signals for a high-dimensional synthesis engine, but its behaviour is constantly influenced and steered by the performer’s actions and auditory feedback (Caramiaux et al., 2014; Van Nort et al., 2013). This creates a scenario where agency is shared and negotiated between the human artist and the machine demonstrating intelligent behaviour (Gioti et al., 2022), fostering emergent musical moments characteristic of improvisational practice (Borgo, 2006). Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th 2 Methods: System Design and Interaction 2.1 Gestural Input: Leapmotion Hand Tracking Hand gestures provide the primary input for the system. A Leap Motion controller captures spatial data about the performer’s hands, including position, orientation, finger angles, and velocity. This high-dimensional data stream offers significant expressive potential compared to traditional controllers but also presents challenges for direct mapping to synthesis parameters (Leeuw, 2021). A Max/MSP patch processes this raw data, extracting relevant features suitable for input into the machine learning model. The core of the interaction logic resides in an MLP implemented in the patch. Unlike predefined mapping functions, the MLP learns relationships between the processed gestural features and NRPN number value arrays representing the internal state of the synthesizer. During performance, the performer’s gestures provide input vectors to the trained MLP. The MLP’s output layer then generates predictions for the target synthesizer parameters. While this prediction process is deterministic, the MLP introduces a layer of abstraction and nonlinearity, transforming gestural inputs into hardware control signals based on a learned internal representation (Fiebrink and Caramiaux, 2016; Esling et al., 2019). The performer influences the training data for the MLP through their movements, effectively teaching the system preferred gesture-state associations or exploring the emergent possibilities of the learned mapping. 2.2 Sound Synthesis Hardware The target for the MLP’s predictions is a hardware sound synthesis platform developed by Destiny+. The key aspect is that the hardware possesses a complex internal state space, involving numerous parameters that interact in non-trivial ways, making direct manual control challenging during improvisation. Moreover, the physical components themselves only allow to modulate four of the parameters at once. The MLP serves to navigate this complexity, translating changes in the performer’s hand positions into meaningful changes within the synthesis engine, drawing parallels with control strategies for modular systems (White, 2022; Carey, 2023). 2.3 Improvisational Interaction: Listening and Spatial Exploration The performance itself is an improvisation based on navigating the possibilities afforded by the system. The performer does not interact with the MLP or the hardware through explicit programming or parameter tweaking during the performance. Instead, interaction occurs through continuous gesture and, crucially, deep listening (Caramiaux et al., 2014; Van Nort et al., 2013). By moving their hands in the Leapmotion’s sensing space, the performer explores the learned mapping function. The sonic output from the hardware synthesizer provides immediate auditory feedback. This feedback informs the performer’s subsequent gestures, creating a closed loop. The exploration is spatial: different regions or types of movement correspond to different sonic behaviours as interpreted by the MLP and realised by the synthesizer. The performer learns to associate gestural motifs with sonic outcomes, guiding the improvisation through this embodied, listening-based exploration of the nonlinear, AI-mediated control space. 3 Relation to Practice Field The use of machine learning as a creative tool aligns with growing interest in AI applications for music generation, control, and interaction (Fiebrink and Caramiaux, 2016; Sturm et al., 2019; Knees et al., 2019). Specifically, employing an MLP for real-time gesture-to-synthesis mapping contributes to research on intelligent performance systems and hyperinstruments, where technology extends the performer’s expressive capabilities (Palacio-Quintin, 2017; Leeuw, 2021). The focus on learning mappings from interaction rather than pre-defining them resonates with approaches that emphasize data-driven and adaptive interfaces (Roma et al., 2019; Fasciani and Wyse, 2012). The improvisational nature of the piece connects to traditions of free improvisation and live electronic music performance (Borgo, 2006; Collins et al., 2003). The system design, emphasizing a feedback 2 loop between performer action, system response, and performer perception, reflects concepts of embodied cognition and enaction in musical interaction (Caramiaux et al., 2014). The exploration of complex mappings and potentially unpredictable sonic behaviours generated by the interaction between gesture, AI, and hardware synthesis can be related to practices involving modular synthesizers and complex systems, where emergent behaviour is often embraced (White, 2022; Slater, 1998; Carey, 2024). Furthermore, the work implicitly probes questions of agency in human-computer interaction (Gioti et al., 2022; Landgraf, 2018). By embedding an adaptive ML component within the control loop, the system moves beyond a simple tool paradigm towards a more collaborative model, where the "instrument" actively participates in shaping the musical outcome based on the performer’s input. 4 Conclusion Cybernesis presents a performance system where human gesture and ML algorithms collaborate in the real-time control of patchable sound synthesis hardware. By utilizing a Leap Motion controller and a multilayer perceptron, the work facilitates an improvisational exploration of complex mappings, navigated through embodied interaction and attentive listening. This approach highlights the potential for ML not just as a tool for automation and generation, but as a dynamic partner in creative expression, fostering emergent musical structures through a tight feedback loop between the artist and the computer. The piece aims to contribute to the ongoing discourse on ML in music creativity, interactive system design, and the evolving nature of improvisation in the age of intelligent technologies. Future work could involve exploring different ML architectures, incorporating more sophisticated feedback mechanisms within model, and investigating the long-term co-adaptation between performer and system. References David Borgo. Sync or Swarm: Musical Improvisation and the Complex Dynamics of Group Creativity. In Lecture Notes in Computer Science, January 2006. doi: 10.1007/11780274_ 1. URL https://www.researchgate.net/publication/221350614_Sync_or_Swarm_ Musical_Improvisation_and_the_Complex_Dynamics_of_Group_Creativity. Baptiste Caramiaux, Jules Françoise, Norbert Schnell, and Frédéric Bevilacqua. Mapping Through Listening. Computer Music Journal, 38(3):34–48, September 2014. ISSN 0148-9267. doi: 10.1162/COMJ_a_00255. URL https://ieeexplore.ieee.org/abstract/document/ 6899813. Conference Name: Computer Music Journal. Benjamin Carey. Ergodynamics of the Open Machine: Material Engagement and Modular Synthesis Performance Practice. Contemporary Music Review, 42(3):376–390, 2023. doi: 10.1080/ 07494467.2023.2277563. URL https://doi.org/10.1080/07494467.2023.2277563 . Publisher: Routledge _eprint: https://doi.org/10.1080/07494467.2023.2277563. Benjamin Carey. Metastable Inventions: Simondonian concretisation and technical invention in modular synthesis practice. Organised Sound, 29(3):315–326, 2024. doi: 10.1017/S1355771824000128. Nick Collins, Alex McLEAN, Julian Rohrhuber, and Adrian Ward. Live coding in laptop performance. Organised Sound, 8(3):321–330, December 2003. ISSN 1469-8153, 13557718. doi: 10.1017/S135577180300030X. URL https://www.cambridge.org/core/ journals/organised-sound/article/abs/live-coding-in-laptop-performance/ 08F42B84BBCA427C345030481A3DDA0D#access-block. Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, and Axel Chemla-Romeu-Santos. Universal audio synthesizer control with normalizing flows, July 2019. URL http://arxiv. org/abs/1907.00971. arXiv:1907.00971 [cs]. Stefano Fasciani and Lonce L. Wyse. Adapting general purpose interfaces to synthesis engines using unsupervised dimensionality reduction techniques and inverse mapping from features to parameters. In International Conference on Mathematics and Computing, 2012. URL https: //api.semanticscholar.org/CorpusID:35264419. 3 Rebecca Fiebrink and Baptiste Caramiaux. The Machine Learning Algorithm as Creative Musical Tool, November 2016. URL http://arxiv.org/abs/1611.00379. arXiv:1611.00379. Artemi-Maria Gioti, Aaron Einbond, and Georgina Born. Composing the Assemblage: Probing Aesthetic and Technical Dimensions of Artistic Creation with Machine Learning. Computer Music Journal, 46(4):62–80, December 2022. ISSN 0148-9267. doi: 10.1162/comj_a_00658. URL https://doi.org/10.1162/comj_a_00658. Andy Hunt, Yo Dd, Marcelo Wanderley, and Matthew Paradis. The Importance of Parameter Mapping in Electronic Instrument Design. Journal of New Music Research, 32, May 2002. doi: 10.1076/jnmr.32.4.429.18853. Peter Knees, Markus Schedl, and Rebecca Fiebrink. Intelligent music interfaces for listening and creation. In Companion Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19 Companion, pages 135–136, New York, NY, USA, March 2019. Association for Computing Machinery. ISBN 978-1-4503-6673-1. doi: 10.1145/3308557.3313110. URL https://dl.acm.org/doi/10.1145/3308557.3313110. Edgar Landgraf. Improvisation, posthumanism, and agency in art (Gerhard Richter painting). Liminalities, 14(1):207–222, 2018. Publisher: Liminalities. Hans Leeuw. Virtuoso mapping for the Electrumpet, a hyperinstrument strategy. In International Conference on New Interfaces for Musical Expression, June 2021. doi: 10.21428/92fbeb44. a8e0cceb. URL https://nime.pubpub.org/pub/fxe52ym6/release/1. Cléo Palacio-Quintin. 2008: Eight Years of Practice on the Hyper-Flute: Technological and Musical Perspectives. In Alexander Refsum Jensenius and Michael J. Lyons, editors, A NIME Reader, volume 3, pages 335–351. Springer International Publishing, Cham, 2017. ISBN 978-3-319-47213-3 978-3-319-47214-0. doi: 10.1007/978-3-319-47214-0_22. URL http: //link.springer.com/10.1007/978-3-319-47214-0_22 . Series Title: Current Research in Systematic Musicology. Gerard Roma, Owen Green, and Pierre Alexandre Tremblay. Adaptive mapping of sound collections for data-driven musical interfaces. In Marcelo Queiroz and Anna Xambó Sedó, editors, Proceedings of the International Conference on New Interfaces for Musical Expression, pages 313–318, Porto Alegre, Brazil, June 2019. UFRGS. doi: 10.5281/zenodo.3672976. URL http://www.nime. org/proceedings/2019/nime2019_paper060.pdf. Dan Slater. Chaotic Sound Synthesis. Computer Music Journal, 22(2):12–19, 1998. ISSN 0148-9267. doi: 10.2307/3680960. URL https://www.jstor.org/stable/3680960 . Publisher: The MIT Press. Bob L Sturm, Oded Ben-Tal, Úna Monaghan, Nick Collins, Dorien Herremans, Elaine Chew, Gaëtan Hadjeres, Emmanuel Deruty, and François Pachet. Machine Learning Research that Matters for Music Creation: A Case Study. Journal of New Music Research, 48(1):36–55, 2019. doi: 10.1080/09298215.2018.1515233. URL https://hal.science/hal-03278041 . Publisher: Taylor & Francis (Routledge). Pierre Alexandre Tremblay, Gerard Roma, and Owen Green. Enabling Programmatic Data Mining as Musicking: The Fluid Corpus Manipulation Toolkit. Computer Music Journal, 45(2):9–23, June 2021. ISSN 0148-9267, 1531-5169. doi: 10. 1162/comj_a_00600. URL https://direct.mit.edu/comj/article/45/2/9/111383/ Enabling-Programmatic-Data-Mining-as-Musicking-The. Doug Van Nort, Pauline Oliveros, and Jonas Braasch. Electro/Acoustic Improvisation and Deeply Listening Machines. Journal of New Music Research, 42(4):303–324, December 2013. ISSN 09298215. doi: 10.1080/09298215.2013.860465. URL https://doi.org/10.1080/09298215. 2013.860465. Publisher: Routledge. Alex White. Unstable Structure: The improvising modular synthesiser. Organised Sound, 27(2):182– 192, August 2022. ISSN 1355-7718, 1469-8153. doi: 10.1017/S1355771821000595. URL https://www.cambridge.org/core/journals/organised-sound/ article/unstable-structure-the-improvising-modular-synthesiser/ CD092D079B056EAEF1FF9CB0D083EEFE#. 4