Noise through to twos and sevens: creating audiovisual artworks from artificial neural networks' processing data

Noise through to twos and sevens: creating audiovisual artworks from artificial neural networks’ processing data Tanguy Pocquet Department of Music University of Manchester Bridgeford St, Manchester M13 9PL [email protected] Abstract This piece is the result of the sonification and visualization of an artificial neural network’s processing data. At its core, it is an attempt at making art from artificial intelligence with a non-generative approach, leaving all creative agency to the artist; at exploring structures inherent to the operation of neural networks; at commenting on the nature of these ubiquitous, yet notoriously unintelligible algorithms. It does so by focusing on some of the network’s internal data streams during training, looking at their gradual movement from noise to order, and mapping this information onto sound and visual materials and processes derived from ‘glitch’ and drone traditions, that offer interesting aesthetic parallels to this evolution of the data. 1 Technical approach and motivations Artificial neural networks (NNs) are commonly known as ‘black box algorithms’. Extensive work has been done on the need to remedy the fact that, generally, “no information is provided about what exactly makes [neural networks] arrive at their predictions” (Samek et al., 2017), and countless approaches have been taken to achieve this (Adadi and Berrada, 2018; Ali et al., 2023). From a technical standpoint, this work situates itself in this search for means to make artificial intelligence (AI) more ‘interpretable’ (providing a “level of understanding of how the underlying technology works” (ISO/IEC, 2020)). Rather than use the product of a trained neural network (a generative approach), it looks at the process that enables this output, at data and metrics from training, when billions of parameters are iteratively tweaked, until the desired output is reached for any given input. These data streams (common performance metrics such as loss and accuracy; and synthetic metrics taken from the generated data such as entropy averages of the samples, similarity between samples, RMS values, edge intensity, etc.) are then mapped onto sound parameters (various granular synthesizers’ parameters, sample lengths, overdrive, overall noise, etc.) and visuals. Beyond offering some insights into how NNs operate, the hope is to present an alternative to generative approaches in the making of artworks based on — or commenting on — AI. This piece uses a generative adversarial network (GAN), trained on the MNIST dataset of images of handwritten digits. The use of as commonplace and simple a dataset as possible felt essential to keep the focus on the process, on the evolution of the data presented over the course of the training process. This piece illustrates how GANs train from noise to order, showcasing various possible intermediary structures. The non-teleological approach is made clearer in the differences between the priorities of most NN designers, and that of this work. The latter is to create aesthetically and narratively interesting results, including through the presentation of sub-optimal runs, the framing of inaccuracies, or the intentional push towards inefficiency. The idea is not to create a particularly efficient GAN, but to demonstrate Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th how it trains, and sometimes fails. Some of the runs displayed in the piece are examples of mode collapse, underfitting, or outright random output from the network, all of which are typical failures from a network design perspective. The stance taken being that there is a lot of understanding of these algorithms to gain from seeing them fail. This approach ties in with the main aesthetic currents that the musical and visual sides of the work are indebted to: ‘post-digital’ and ‘glitch’ traditions. 2 Aesthetic and artistic context While technically algorithmic sonification, this work links to systematic music as a whole, algorithmic music more specifically, and even more specifically AI approaches to algorithmic music (Todd, 1989). It also links to other examples of NNs being repurposed for non-generative music-making (Tudor, 1995). In essence, reflecting some of the technical priorities detailed above, this work is perhaps an illustration of the view that “The point of art is detritus” (Weiner, 2012); it focuses on processes, and on what are normally byproducts, extraneous material. Sonically, it owes a lot to ‘glitch’ music, a subgenre that harnesses “the ‘failure’ of digital technology [. . . ]: glitches, bugs, application errors, system crashes, clipping, aliasing, distortion, quantization noise, and even the noise floor of computer sound cards are the raw materials composers seek to incorporate into their music” (Cascone, 2000). More broadly, it links to the post-digital aesthetic, which “takes digital to be the pinnacle of audio fidelity (technology driven by the ultimate desire for perfect clarity and the elimination of noise) and situates itself after: after the revolution, amidst the rediscovery of noise through digital malfunction” (Haworth, 2016). This link between the information being sonified (byproducts, detritus) and the sound they effect (sounds of errors, digital malfunction, clicks, and noise) is central to the project. In many data-driven pieces, there seems to be a need for aural representations of parsing; scanning; filtering; grouping; ordering; etc., all concepts we associate with data-handling. This can take the form of sounds associated with barcode scanners (Ikeda, 2008, 2005); hard drives’ read-and-write heads (Panacea’s Synecore); electromagnetic glitch/clicks/interference (Ikeda, 1996; SND, 2010); a human voice reading out litteral data/numerical information (alva noto, 2008). This extends to visual/semantic representations of data: barcodes (Ikeda, 2005); binary numbering (Ikeda, 2008); references to coding (SND, 2000); mathematical concepts (Ikeda, 1996); or a self-referential focus (the music itself as the data) (Goem, 2013; SND, 2000; Cascone et al., 2004; Cyclo, 2001). This apparent need for links to material processes in aural representations of data that is — by nature — immaterial, is also something this piece attempts to address. As Eno observed, the use of these types of sounds, and of their semantic implications, is born of a need for materiality in electronic music: “after the loss of materiality with the appearance of digital music, one might see the advent of music evoking imagined but specific and tangible media as caused by a need to rematerialise sound” (Weium and Boon, 2013). This piece might lead to one of these imagined media, linked to, but distinguishable from, the actual algorithmic processes it is driven by. This piece also uses NNs as another creative tool for sound structuring (Haworth, 2016). It explores the idea that the vast amounts of data generated by modern technology form “a canvas for the contemporary artist” (Lynch and Paradiso, 2016), leading to a “database aesthetic” (Vesna, 2007). Finally, it displays materials linked to instrumental and electronic drone traditions (Davachi, 2023; Malone, 2019; Akbro, 2017; Radigue, 2018; Conrad, 2017). A lot of the source materials are organ recordings, of various drones and improvisations. The contrast between the highly synthetic glitch sounds, and highly organic and inherently human sounds of a church organ forms an interesting parallel with the GAN’s gradual reproduction of handwritten digits, with the move from highly abstracted, random numerical data, to an intelligible, human-looking output. References Adadi, A. and Berrada, M. (2018). Peeking inside the black box; a survey on explainable artificial intelligence. 6:52138–52160. 2 Akbro, E. (2017). For organ and brass. Subtext recordings, London. Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J., Confalonieri, R., Guidotti, R., Del Ser, J., Diaz-Rodriguez, N., and Herrera, F. (2023). Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. 99(101805). alva noto (2008). unitxt. Raster-Noton, Chemnitz. Cascone, K. (2000). The aesthetics of failure: "post-digital" tendencies in contemporary computer music. 24(4):12–18. Cascone, K., Kahn, J., and steinbruechel (2004). ATAK004. ATAK, Tokyo. Conrad, T. (2017). Ten years alive on the infinite plain. Superior Viaduct, San Francisco. Cyclo (2001). .NOTON, Chemnitz. Davachi, S. (2023). Long gradus: arrangements. Late music, Los Angeles. Goem (2013). Stud stim. Korm digitaal, Nijmegen. Haworth, C. (2016). ‘all the musics which computers make possible’: questions of genre at the prix ars electronica. 21(1):15–29. Ikeda, R. (1996). +/-. Touch, London. Ikeda, R. (2005). dataplex. Raster-Noton, Chemnitz. Ikeda, R. (2008). test pattern. Raster-Noton, Chemnitz. ISO/IEC (2020). Software and systems engineering, software testing, part 11: Guidelines on the testing of AI-based systems. Lynch, E. and Paradiso, J. (2016). SensorChimes: Musical mapping for sensor networks. Brisbane, Australia. Malone, K. (2019). The sacrificial code. Ideologic organ, Paris. Radigue, E. (2018). Occam ocean I. Shiiin, Paris. Samek, W., Wiegand, T., and Müller, K.-R. (2017). Explainable artificial intelligence; understanding, visualizing and interpreting deep learning models. SND (2000). stdio. Mille plateaux, Frankfurt. SND (2010). makesnd cassette. Mille plateaux, Frankfurt. Todd, P. (1989). A connectionist approach to algorithmic composition. 13(4):27–43. Tudor, D. (1995). The neural network synthesizer. Vesna, V. (2007). Database aesthetics - art in the age of information overflow. University of Minnesota press, Minneapolis. Weiner, L. (2012). Lawrence weiner interviewed by hans ulrich obrist. Weium, F. and Boon, T. (2013). Material culture and electrtonic sound. Artefacts: studies in the history of science and technology. Smithsonian institution scholarly press, Washington, D.C. 3