Opening Up Sound Space

Opening Up Sound Space Alain Bonardi1[0000-0001-7647-8156] and Paul Goutmann1[0009-0009-4889-0579] 1 CICM/MUSIDANSE, University of Paris 8, 2, rue de la Liberté, 93526 Saint-Denis Cedex 02 (France) [email protected] [email protected] Abstract. The widespread availability of equipment and processes for creating sound spaces in the digital audio field has been accompanied by a musical and technical convergence towards certain ‘mainstream’ spatialization processes based on the acoustic modeling of point sources in a reverberant environment. But the question arises to understand how far this approach of sound spatialization embraces and grasps sound space both conceptually and physically. In fact, a number of approaches, such as spatial sound synthesis and spatial sound processing we demonstrate in the paper, have shown great generative and expressive potential. These approaches led to experiment new representations and interfaces that go beyond the usual Euclidean framework for manipulating sound spaces. The generalization of this research led us to design the ERC Advanced Grant G3S project, which combines machine learning and sound spatialization, and which we outline here, in terms of generativity, operative representation, description of spatiality and exploration interfaces. Keywords: Spatialization, Sound Space, Spatial Audio Processing. 1 Where Are We With Sound Space? The presence of electronically-produced sounds in everyday life has become very important in recent years. When we describe these sounds, we generally refer to their frequency content and temporal unfolding. We often forget the spatial dimension of electronic sounds, which is absolutely essential. Since the 1990s, several software programs implementing sound spatialization models have been created and shared by communities of composers, computer music designers and sound engineers. Sound spatialization is a rapidly expanding field on many levels, stimulated by a wide range of industrial and artistic applications. In recent years, we have seen a general trend towards the appropriation of spatialization by creators, some of whom no longer hesitate to equip their personal home studios with five or more loudspeakers, going beyond simple stereo. Based on Head Related Transfer Functions (HRTF), binaural technology [27] enables headphones to simulate sensations of 3D immersion. Many concert halls, cinemas and studios are now equipped with multipoint diffusion systems for sound spatialization. More advanced installations such as acousmoniums, 2D octophonies, 3D domes or WFS loudspeaker Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 347 2 A. Bonardi and P. Goutmann antennas are generally only available in institutions dedicated to creation, research and teaching. The field of space sound is stimulated by a wide range of industrial and artistic applications in a context of international standardization. Object-based audio 1 is gaining ground not only in the creative world, but also in broadcasting. For example, the ADM 2 (Audio Definition Model) standard used in the audiovisual industry allows to add metadata concerning the spatiality of each audio channel and each “object” 3 . For several years now, the industrial market for 3D audio, which offers listeners complete sound immersion, has been expanding rapidly worldwide. 2 Spatialization and sound space For composers and sound creators, this sociological, technical and industrial convergence to more and more formatted processes of spatialization as described above leads to question the relationship between these processes and the sound space as it is generated and experienced [31]. Does sound spatialization embrace and grasp sound space both conceptually and physically? In this question, we use the verbs “embrace” and “grasp” in their multiple meanings. For the verb “to embrace”, we think first of its physical meaning, of holding someone closely in one’s arms: in a way, sound space should embrace the listeners by holding them in “its arms”. The other meaning of “to embrace” is the enthusiastic acceptance of something, in our case, we can say that more and more sound creators have embraced the converging and standardized technologies of spatialization. The first meaning of the verb “to grasp”, which is “to seize and hold firmly” also refers to physical action as “to embrace”, whereas its second meaning refers to understanding: how far can we understand and handle sound space through the techniques of mainstream spatialization? In short, embracing and grasping sound space via spatialization raises questions of physical and intellectual apprehension of the latter, and of the homogenization of creative practices. There are two possible answers to our previous question: “Does sound spatialization embrace and grasp sound space both conceptually and physically?”. They are “yes if…” and “no if…”. 2.1 Yes if… We should answer “yes” if we consider sound space on the basis of the physico-acoustic modeling of a space conceived in an additive way: we add and position (or move) punctual sound sources, immersed in a reverberant environment. These are manipulated via graphical interfaces that call on Euclidean representation models, taking the sound source reduced to a point in three-dimensional space as the fundamental object. These 1 This practice consists in considering each sound element as a distinct “object” with its own attributes. 2 https://adm.ebu.io/background/what_is_the_adm.html 3 For each object, its nature, spatial position, movement and other parameters. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 348 Opening up sound space 3 spatialization techniques are closely related to models used in psychoacoustics apprehending spatiality in terms of point source location [3] [28] and room acoustic qualities [2]. The spatialization interfaces represent 3D space and the spatial configuration of the sources must correspond to the final sound result. In practice, spatial audio is generally produced in post-production using a workflow that separates the creation of mono sounds from their positioning in space. The Spat developed at Ircam [10] represents an archetype 4 of this way of composing spatialization from sources and a model of the room where the sound spreads through its reverberation. The aim is also to achieve realistic characteristics by being able to simulate, for example, air absorption or a specific room through convolution. Another important feature of Spat is its universality in terms of point spatialization, since the environment integrates numerous techniques for panning individual sources (including VBAP 5 , LBAP 6 , VBIP 7 , KNN 8 ) as well as High-Order Ambisonics [14] (HOA, binaural) or WFS 9 . These environments are associated with control interfaces based on the placement of point sources [9] in a virtual space (2D or 3D). More recently, the use of virtual reality has been experimented [13] to control spatialization and offer an immersive rendering of the sound result, by integrating its calculation into the overall rendering, following the position of the head. It makes possible to place sound sources in VR space, whether for new types of music diffusion [22] [34] or electroacoustic creation [39]. 2.2 No if… On the contrary, we should answer “no” if we consider that sound space goes beyond spatialization based on punctual sound sources. We should then open our eyes and ears to other approaches, admittedly less widespread than mainstream spatialization methods considering sounds as points in space: the spatial qualities of sound in music cannot be reduced to location, movement and room acoustics. In the contemporary music of the second half of the 20th century, the notion of sound mass appeared in written music (without electronics) as a new conception of composition in Edgar Varèse (Hyperprism, 1923) or Stockhausen (Gruppen, 1958). This conception influenced electroacoustic creation, as shown [12] by the example of Xenakis' Concret PH (1958). In computer music, the notion of the sound cloud has been applied in various ways. A first approach consists of working with a very large number of sources to obtain a surface or volume composed of these sources, under the assumption of boundary crossing. Sound synthesis can be envisaged spatially by composing sound from a large number of partials, each considered as a point source [38]. Some research4 Among a large number of environments, for instance SpatGris [25]. 5 Vector-Base Amplitude Panning. 6 Layer-Based Amplitude Panning. 7 Vector-Base Intensity Panning. 8 K-Nearest-Neighbor amplitude panning. 9 Wave Field Synthesis. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 349 4 A. Bonardi and P. Goutmann ers are also introducing spectral spatialization into the realm of frequency-domain processing techniques [24] and experimented with the diffusion of grains at different locations and depths in space [32]. Barry Truax [35] emphasized the relationship between micro-temporal operations, timbre and spatiality, making it possible to increase the apparent size of a sound 10 . In the 1990s, researchers such as Kendall [23] began to investigate the use of decorrelation 11 of audio signals to create spatial imagery. Hagan [21] developed the notion of textural composition in space using uncorrelated and distributed signals thanks to randomization. Sèdes [33] then focused on the musical potential of decorrelation in sound creation. All this work has proved its relevance because the resulting musical and sound creations sound good, perceptually validating the research carried out. The general ideas behind all these approaches are: • We can interweave the operations of sound processing or synthesis with those of constructing spatiality. In this case, spatial audio is no longer produced in post-production but the sound is natively generated as spatial. • Manipulating and generating sound space does not necessarily require Euclidean spatial representations, but should open to other ones. • Spatial rendering can combine different models that will be controlled by our perception. • We could consider mainframe spatialization as particular methods among other ones to generate sound space. 3 Exploring operation intrications and their representations This section examines two case studies of original approaches to spatial audio: Dans la Nef de nos songes and Émergences. In the first one, a recursively expanded bell model generates 121 independent channels whose ambisonic mappings yield radically different auditory spaces. In the second one, space is treated not as a container but as a morphology to be textured, stratified, and transformed. These works propose an operative view of space-as-substance, where sound and spatial form co-emerge. 3.1 Spatial sound synthesis: Dans la Nef de nos songes This research started with the commission of a sound installation to Alain Bonardi in 2018 at the Hôtel-Dieu in Tonnerre (France), which is a large medieval building, whose main room (1600 square meters) is dedicated to contemporary exhibitions. The sound installation, Les Songes de la nef, was shown in summer 2018. We started from the bell model in Risset's Introductory Catalog of Computer Synthesized Sounds (1969), based on the combination of 11 harmonic and inharmonic partials, and extended this model 10 Truax described a tool able to dispatch flows of granularized sounds to various loudspeakers thanks to a 8 x 8 matrix. 11 Process by which a sound is transformed into multiple, slightly different but identical-sounding instances. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 350 Opening up sound space 5 recursively to build a bell with 11 x 11 = 121 partials [5]: each partial of the original bell becomes the virtual fundamental of a new bell. This operation can be understood as a multiplication of chords/spectra, as used by Boulez for instance [7]. In our case, the 121 partials come from the multiplication of the spectrum of 11 notes by itself. This approach locally preserves the features of the sound of the bell while generating a complex sound composed of numerous partials [6]. If we consider the 121 sounds as 121 channels (without mixing them to a single channel), how can we spatialize them? In Les Songes de la nef, we organized a simple mapping that would change at every section. After this sound installation, we decided to explore more systematically the spatial synthesis of sound ; this research led to a new electroacoustic piece, Dans la Nef de nos songes. We built a generic Max environment to explore the possible mappings of the 121 channels to the circular or spherical harmonics of the ambisonic model (having chosen an ambisonic order in 2D or 3D up to order 3). The core of this environment is a matrix (see Figure 2) enabling the dispatching of the channels to the ambisonic harmonics and a Javascript program enabling to sort the partials in different manners: by sets of 11 partials (bells at level 2), by frequencies, by onset dates, either in ascending or descending order. To manage onset dates, we have introduced attack offsets, so that the sounds are not all triggered at the same time, but slightly staggered, as if arpeggiated over a certain duration. The output of the matrix is connected to an ambisonic decoder at a given order (up to 3) in 2D or 3D. Figure 1 shows the processing chain. Fig. 1. The processing chain for spatial synthesis. Once the synthesis parameters are set (global fundamental, maximum delay of onset), the sound content generated as input of the matrix remains the same; we could record it as a 121 track file. But a very interesting fact occurs here: from one matrix to another, the result sounds different (and sometimes very different), changing the spectral centroid, merging or separating partials. In a way, the sound space of the result is created by the 121 channel content itself, by a combination of amplitude and precedence effects. This provides interesting ways to investigate the intrication between sound creation and its spatial life. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 351 6 A. Bonardi and P. Goutmann One of the difficulties that immediately emerges is that of controlling the generated sound space, which is far from intuitive, since the interface does not represent a Euclidean space in which the user can act, but a singular object, the distribution matrix with no immediate connection to spatial properties. Other ways of distributing partials are undoubtedly possible, as we have shown in [4]. Fig. 2. An instanciation of the dispatching matrix, by descending order of onset dates. 3.2 Spatial sound processing: Émergences This reflection originates from the compositional work in 3D ambisonics carried out in the piece Émergences, in which spatial morphologies were treated as the core musical interest. The composition of Émergences was initiated in 2022 by Paul Goutmann, under the auspices of a doctoral thesis in research-creation, which was funded by the ArTeC graduate school 12 . This approach serves two primary functions: firstly, as a means of evaluating the software and theoretical tools developed as part of the thesis, and secondly, as a means of contributing to the implementation of research directions. In this process, music creation and software development (both digital signal processing and graphical user interface) interacted with constant feedback effects. 12 Émergences was created at the MSH PN for the JIM 2023, a binaural version is available in appendix of the article [21]: https://hal.science/hal-04084391v1. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 352 Opening up sound space 7 In Émergences, the spatial characteristics were not confined to position, movement, or room response, but were instead explored as morphologies – structured, multi-scale entities that could be composed and articulated. The concept of "spatial morphology" has been adopted from Horacio Vaggione's morphological approach [36] [37], in which sound forms emerge from transformational operations applied across multiple temporal scales. In this piece, the notion of space is approached not simply as a container or projection field, but rather as an operative, malleable dimension. The study focuses on three types of spatial morphology: spatial texture, spatial envelope and stratification [19]. Instead of composing textures in space, an attempt was made to texture the space itself. The distinction here is subtle yet crucial: rather than organizing sound sources to produce a texture, we have manipulated original decorrelation processes in HOA and superimposed multiple multichannel recordings to generate a sense of spatial variation. The attributes selected to generate spatial texture included density (number and interrelation of layers), spectral distribution (via filtering), depth sensation, agitation (rate of variation), and verticality (spatial stratification). For now, those dimensions are only experimented with a composition point of view and the salience of each dimension is not well assessed. Nevertheless, these offered researchers "handles" on the space-assubstance. As illustrated in Figure 3, the initial layer processing chain uses seven parallel decorrelation modules, which are controlled by multiple graphical user interfaces. Fig. 3. Process chain on the first layer with seven decorrelation modules controlled by abc.dtd.ui GUI. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 353 8 A. Bonardi and P. Goutmann In this context, 'stratification' is understood to signify the vertical layering of spatial treatments, effectively constituting a multi-tiered approach to three-dimensional space. This approach involves the utilization of multiple horizontally arranged planes, each characterized by distinct processing parameters and spatial behaviours. The objective was not merely to elevate sources, but rather to devise a vertical architecture of sound fields, with each field possessing its own sonic identity and internal dynamics. Each stratum processes the same source material. The lowest two layers use feedback within their decorrelation networks, thereby producing dense textures. The uppermost layer, situated above the listener on the loudspeaker array, was processed with non-feedback decorrelation modules that processed unfiltered impulse sounds. This resulted in sharp broadband impulses that, paradoxically, localized less precisely due to psychoacoustic blur at the zenith [3]. Furthermore, by gradually alternating sound sources between strata, we investigated perceptual effects analogous to vertical motion of spatial "planes" rather than point sources, thereby posing the question: can spatial texture itself move vertically? This challenges conventional spatialization paradigms, emphasizing not only point-source trajectories but also transformations of spatial morphology. This perspective inaugurates new domains for compositional praxis and spatial audio research, positing that space-as-substance constitutes more than a mere metaphor; it is a perceptual reality to which composers can apply their original instruments [20]. 4 Towards new approaches of sound space: the ERC AdG G3S project The present paper opens the 5 year ERC Advanced Grant G3S project submitted by Alain Bonardi, and recently selected in June 2025. G3S stands for “Generative Spatial Synthesis of Sound and Music”, and it precisely refers to the exploration of new approaches of sound space going beyond mainstream spatial audio. During the project, we will systematically generalize the existing research in the various fields related to operations for space generation (to open up sound spaces) and representations to handle them (going beyond the Euclidean representations of 3D). Moreover we will add two new dimensions of research: • the generation of sound spaces using machine learning techniques that will enlarge the generative possibilities in terms of spatiality • the qualitative and quantitative description of sound spatiality that were missing in our previous experiments about spatial sound synthesis and spatial sound processing. 4.1 Context of the G3S project In recent years, approaches to adding spatiality to existing sounds have become an industry challenge for several companies offering spatialization of sound content. Dolby Atmos, for example, offers spatialization solutions for the film and music industries. But the processes involved are not publicly accessible. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 354 Opening up sound space 9 For the time being, artificial intelligence has little to do with these technologies: its meteoric development in the music industry, both in terms of usage and production organization, mainly concerns the generation of tracks in given styles [8]. Sound space is often neglected in generative approaches to music, to the detriment of melody, harmony and rhythm [26]. Human-machine interaction, between human musicians and computers, is addressed by [15], who emphasizes the importance of machine learning in bringing new ways of thinking about creation. Environments using machine learning for the generation of new sound timbres [18] or human-machine co-improvisation [16] [1] only manipulate and generate sounds in stereo at best. However, machine learning systems are beginning to be used at different stages of the spatial audio pipeline [11]. For instance, in order to spatialize the soundtracks of videos initially in mono or stereo, automatically relating objects shown in the image to the corresponding sound in the audio [30]. Other applications are appearing that automatically convert mono sound from a 360° video camera into “spatial” audio [29]. Another use that is emerging in the music industry is the automatic separation of musical parts mixed in a stereo file to enable them to be spatialized. Some companies, such as AudioShake, focus more specifically on separation, while others, such as Master Channel, offer solutions that go as far as automatic spatialization, inspired by the practices of professional sound engineers. Ircam start-up Amplify also offers solutions for moving from stereo to ‘spatial’ sound by separating voices and automatically spatializing them 13 . Although very interesting in sound engineering and creation, these various attempts are limited to a Euclidean approach to sound space. Some advances have come from artistic creation: composer-researcher Aaron Einbond and his colleagues at Ircam [17] have proposed two creative experiments to spatially radiate sound captured from acoustic instruments in real time. Captured 3D radiations of orchestral sounds were considered as models and training sources for a machine learning device then deployed in real time. 4.2 Research objectives of the G3S project While the industrial markets for 3D audio and artificial intelligence in music are expanding rapidly worldwide, the spatial qualities of sound are still an unthought-of aspect of AI. In practice, spatial audio is generally carried out in post-production and is usually confined to the spatialization of sound and the acoustic modelling of rooms. The aim of the G3S project is to propose new ways of creating, modelling and analyzing sound by natively integrating its spatial dimension. We want to extend the theoretical and practical approach to spatial diffusion, whether with loudspeakers or headphones, by using the generative capabilities of artificial learning. AI will enable us to generate new sound spaces from all existing approaches: this is what we call generative spatial synthesis. This will enable us to move beyond the dominant paradigm of point-based sound spatialization, which will become a special case of a more general approach. Rather than thinking of sound space solely in terms of geometry in a Euclidean space by composing in space, our approach will lead to composing space according to the 13 https://www.ircamamplify.io/product/spatial-audio Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 355