tinySounds: for voice and musebot ensemble

tinySounds: for voice and musebot ensemble Arne Eigenfeldt School for the Contemporary Arts Simon Fraser University, Canada [email protected] Abstract An ironic work in which tiny sounds – quiet noises made by the human voice that are barely audible – serve as an input for a noisy and exuberant musebot ensemble that autonomously responds, accompanies, and argues with the live input. Musebots are intelligent musical agents that decide how to respond to their environment – and each other – on their own, based upon their internal beliefs, desires, and intentions. 1 Detailed Program Notes Machine learning algorithms are wonderful for sifting through data and discovering relationships; more challenging is how these algorithms can be used for generation. It isn’t that difficult, for example, to train a system to provide similar sounds for a database, given a live sound. But what’s the artistic interest in that? Similarly, it isn’t that difficult to extract live performance information from an improvising musician – activity level, general frequency range, timbre – so that the system responds likewise. But, again, reactive systems lose interest fairly quickly. I find it much more interesting when my musebots go off on their own, exploring their own ideas through beliefs they may have formed incorrectly and unintentionally. For that reason, I usually build a lot of ambiguity into my analysis, or provide conflicting information. What happens when one musebot is sure of something, while another is absolutely sure of something else? And what if a third musebot just doesn’t care? In tinySounds, musebots are trained using a neural net on a corpus that has been hand-tagged for valence and arousal measures, as well as pre-analysed for spectral information. However, the correlation between audio features (what the musebots are listening for) and affect (valence and arousal) isn’t direct; in assigning the latter, I may decide that a sound from the corpus is complex and active, but my reasons for doing so may not use the same information as the musebots are provided with. Thus, a musebot may decide that, based upon what it has learned, a live sound is high valence / high arousal, but the listener may perceive it otherwise. This isn’t a flaw in the system; it’s a feature! Lastly, my role as overseer in the musebot ensemble allows me to further disrupt how the musebots apply their knowledge. The corpus is organised semantically (i.e. voice sounds, kitchen sounds, transportation sound, etc.); once a musebot is using a certain subdirectory, it can’t easily switch to another. As a result, its choice of related sound, whether affective or timbral, is limited to what is immediately available to it. If the musebots are frustrated, they haven’t mentioned it to me (yet). 2 Musebots Musebots are pieces of software that autonomously create music collaboratively with other musebots (Bown et. al, 2015). They decide how to respond to their environment – and each other – on their own, based upon their internal beliefs, desires, and intentions. Since 2015, I have used musebots in a variety of artworks (Eigenfeldt 2016, Eigenfeldt 2017, Eigenfeldt 2018, Eigenfeldt and Ricketts 2019), and became their main evangelist. I had been working with agents for almost a decade, and the musebot protocol allowed for a consistency that Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th kindled opportunities to adapt and reuse agents outside of their original creative work. Musebots lack direct interactivity, as they are autonomous. As a result, my role during their performative actions is less as a performer or conductor, then as a critical listener. This is particularly the case during the long periods of fine tuning every musebot ensemble, which entails a great deal of listening and note-taking in order to discern creative autonomy from buggy code. Musebots are not straightforward reactive processes; instead, they have their own beliefs (in this case, the incoming analysis data), desires, and intentions. They will happily play on their own, or they may react very closely to the live performance; more often then not, they will offer their own “reinterpretation” of the live performance, with individual reactions to the analysis data. 2 Context The musebot ensemble in tinySounds is a redeployment of an earlier metacreative system, The Indifference Engine, which is partially described elsewhere (Eigenfeldt, 2014). Live audio is analysed for features: spectral centroid; spectral flux; loudness; activity level (onset detection); and Bark band spectrum. This information is messaged to the audio musebots and an effectsBot. This latter musebot adds effects – delay, pitch shift, time stretch, ring modulation, and distortion – autonomously, based upon its interpretation of the analysis messages. For example, it will switch effects when activity is low, and add more processing when flux is high. The audio musebots – in this case, four instances of tinySoundBot – have access to a large corpus of pre-analysed soundfiles; given a Bark band spectral analysis via the Conductor, the audioBots will attempt to find the closest matching recordings from their available database. The audioBots autonomously begin and end playing based upon incoming messages, including activity and flux, as well as reacting to whether other audioBots are active or not. Audio is generated using a modified version of CataRT (Schwarz, 2007). References Bown, O., Carey, B., & Eigenfeldt, A. (2015) Manifesto for a Muse-bot Ensemble: A platform for live interactive performance between multiple autonomous musical agents. In Proceedings of the International Symposium on Electronic Art, Vancouver. Eigenfeldt, A. (2014) Generating Structure – Towards Large-scale Formal Generation. In Proceedings of the Tenth Artificial Intelligence and Interactive Digital Entertainment Conference, Raleigh. Eigenfeldt, Arne. "Musebots at One Year: A Review." Proceedings of the Musical Metacreation Workshop. 2016. Eigenfeldt, Arne. "Designing Music with Musebots." Proceedings of the Fifth Conference on Computation, Communication, Aesthetics and X. 2017. Eigenfeldt, Arne. Collaborative Composition with Creative Systems. International Symposium on Electronic Art, Durban, 2018. Eigenfeldt, Arne, and Kathryn Ricketts. “Unauthorized: Collaborating with a Performer Collaborating with Creative Systems.” Generative Art Conference, Rome, 2019. Schwarz, D. (2007) Corpus-based Concatenative Synthesis. In IEEE Signal Processing Magazine, 24(2). 2