Semantic Control Over Neurally Synthesized Audio via Latent Disentanglement

Master in Sound and Music Computing Universitat Pompeu Fabra Semantic Control Over Neurally Synthesized Audio via Latent Disentanglement Jed Padoa Supervisor: Lonce Wyse Co-Supervisor: Frederic Font August 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 1 1.2 Objectives.................................. 2 1.3 Structure of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Background 4 2.1 AudioRepresentations ........................... 5 2.2 Latent representations . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.1 Interpretability ............................... 7 2.3 Audioembeddings ............................. 10 2.3.1 Contrastive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 VAEs, GANs, and RAVE 12 3.1 Variational Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.1 MarginalLikelihood ............................ 13 3.1.2 KL-Divergence ............................... 14 3.1.3 Reparameterization Trick . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.4 ConditioningVAEs............................. 15 3.2 Generative Adversarial Networks . . . . . . . . . . . . . . . . . . . . . 16 3.2.1 Architecture................................. 17 3.2.2 Training................................... 17 3.3 RAVE .................................... 18 3.3.1 Architecture................................. 19 3.3.2 Training................................... 20 4 Experiment 23 4.1 Experimentdesign ............................. 23 4.1.1 Dataset ................................... 24 4.1.2 Embedding and Attribute Computation . . . . . . . . . . . . . . . . . 24 4.1.3 Audio Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.4 ModelArchitecture............................. 27 5 Results 31 5.1 Attribute Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 5.1.1 Speed .................................... 31 5.1.2 Surface Material Attributes . . . . . . . . . . . . . . . . . . . . . . . . 31 5.1.3 Plots..................................... 32 5.2 SemanticFaderModel ........................... 33 5.2.1 Reconstruction Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.2.2 Quantitative Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5.2.3 Perceptual Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6 Discussion 37 6.1 ChallengesFaced .............................. 37 6.1.1 PosteriorCollapse.............................. 37 6.1.2 Inconsistent Footstep Tempo . . . . . . . . . . . . . . . . . . . . . . . . 39 6.2 Conclusions ................................. 39 6.3 FutureWork................................. 40 List of Figures 41 List of Tables 42 Bibliography 43 Abstract Advances in deep generative models have made it possible to synthesize high-fidelity audio, yet giving users precise, continuous control over the semantic qualities of the generated sound remains challenging. This thesis tackles the problem by combining variational auto-encoders (VAEs) with a latent disentanglement strategy inspired by Fader Networks [1][2]. In this model the encoder learns a compressed latent space invariant to desired control attributes, allowing for precise control over said attributes before the latent representation is passed to the decoder via a "fader" like mechanism. Attributes are computed via a learned linear regression coefficient trained in a supervised manner on continuous attribute labels derived from a synthetic footstep sound effects dataset. During training, adversarial and reconstruction losses encourage orthogonality between the latent codes and control attributes, ensuring that adjusting one attribute adjusts only the desired content while leaving the rest unchanged. This thesis details the state of the art and core concepts behind the methods used before diving into the details of the implementation. Finally, the results will be presented and discussed. Acknowledgement I would like to express my sincere gratitude to my supervisors Lonce Wyse and Frederic Font as well as the many teachers and colleagues at the MTG who provided me with wisdom and guidance throughout my time there. Chapter 1 Introduction 1.1 Motivation Recent advancements in generative AI have led to a remarkable surge in creative capabilities, enabling the generation of highly realistic images, text, and audio. These models, trained on vast datasets, can produce novel content that is often indistinguishable from human-created works. In the audio domain, this has resulted in models that can synthesize music in various styles, generate realistic human speech, and create a wide array of sound effects. This progress has opened up exciting possibilities for content creation, music production, and accessibility tools, demonstrating a powerful ability to learn and replicate complex patterns from data. Despite these impressive capabilities, a significant challenge remains: the lack of fine-grained, intuitive control over the generated output. While users can often guide generation with text prompts or high-level descriptions, controlling specific, continuous attributes, particularly semantic ones like the emotional intensity of a voice or the constituent materials of a sound effect, is often difficult or impossible. The generation process can feel unintuitive, and minor changes to the input can lead to unpredictable or undesirable changes in the output. This gap between the models’ generative power and our ability to direct it with precision highlights a crucial area for improvement in generative audio systems. 1 8Chapter 2. Background it difficult to objectively assess whether a model has learned a useful structure. Even if a latent space is interpretable, navigating it to create meaningful transformations is not straightforward. The most obvious approach, linear interpolation between two points in the latent space, often fails to produce a perceptually smooth transition, due to uncertainty as to whether or not the path moves through realistic outputs. For audio, this can result in a series of muffled, noisy, or perceptually implausible sounds during a transformation. This problem highlights that the geometry of the latent space is rarely simple or understandable. To generate a sequence of valid intermediate samples in the latent space of a VAE for example, one must find a path that remains within high-probability regions of the learned distribution. Two common approaches to latent space disentanglement for generative models include β-VAE and InfoGAN. The β-VAE introduces a hyperparameter βto the standard VAE objective, encouraging the latent distribution to closely match a factorized prior and thus disentangle independent factors of variation. By penalizing the Kullback-Leibler (KL) divergence more heavily, β-VAEs can learn latent spaces where single dimensions correspond to interpretable attributes. However, if the β term is too large the increased interpretability may come at the cost of reduced reconstruction fidelity [4]. InfoGAN, on the other hand, augments the GAN objective with an information-theoretic regularizer to maximize mutual information between a subset of the latent space and the data, enabling the recovery of interpretable, disentangled factors in an unsupervised manner [5]. Building on these techniques, FactorVAE explicitly encourages independence among latent dimensions by penalizing the "total correlation," a statistical measure of redundancy among random variables. FactorVAE achieves a superior trade-off between disentanglement and reconstruction quality compared to β-VAE, but requires an auxiliary discriminator to estimate total correlation, introducing additional complexity to training [6]. Further innovations such as β-TCVAE and DIP-VAE propose refined regularizations or posteriors, addressing some of the stability and complexity issues present in earlier models. Recent work also questions the reliability of disentanglement metrics, highlighting that even high-scoring models may not guarantee 2.2. Latent representations 9 meaningful or consistent control over generative factors, particularly in complex domains like audio [5]. Thus, while current approaches like β-VAE, InfoGAN, and FactorVAE have significantly advanced the disentanglement of latent spaces, open challenges remain in balancing interpretability, generative quality, and reliable evaluation. Fader Networks Fader Networks aim for controllable generation by learning a latent space where dimensions correspond to semantically meaningful attributes. The concept involves training a generative model such that traversing a specific latent space direction or dimension results in a continuous, predictable change in a particular output attribute (e.g., brightness, style, timbre) while others remain unaffected [1]. This is achieved through an adversarial approach whereby a discriminator network attempts to predict certain features by inspecting the latent space itself. The goal of the model’s encoder is to learn a representation invariant to these features, meaning the encoder ignores them, in order to fool the discriminator. The true feature values are then appended to the latent space, and decoded to produce outputs. This adversarial aspect allows the user to pinpoint exactly which dimension corresponds to the desired control attributes, allowing for controllable generation with regard to the chosen features [1]. F-RAVE, an extension of the generative audio model RAVE, builds upon this concept by learning a mapping between continuous, human-understandable audio descriptors (e.g., spectral centroid for brightness, RMS for loudness) and the latent space of a generative model [2]. This enables finer, more intuitive control over generated sound by directly manipulating these descriptors. The idea behind Fader Networks builds on the broader goal of creating a well-organized latent space where different data aspects are encoded in a disentangled way. Fader Networks offer a promising approach to making generative audio models more controllable and user-friendly by providing intuitive ways to manipulate specific sound characteristics. Their development highlights the increasing demand for more interpretable and controllable generative models in audio, moving beyond realistic sound generation to precise artistic expression and manipulation. 10 Chapter 2. Background 2.3 Audio embeddings Audio embedding is a technique in machine learning that transforms complex audio signals into compact, low-dimensional numerical vector representations [7]. Raw audio, whether as a time-domain waveform or a high-resolution spectrogram, is inherently high-dimensional and difficult for many algorithms to process directly. The primary goal of an embedding is to distill the perceptually and structurally important characteristics of a sound—such as its timbral qualities, pitch contours, and rhythmic patterns—into a dense and meaningful representation [8]. This not only makes the data more computationally tractable but also creates a structured feature space where sounds with similar acoustic properties are positioned closely together, enabling tasks like similarity-based retrieval, classification, and manipulation for creative applications. These embeddings are most commonly learned using deep neural networks, particularly models with an autoencoder architecture. In this framework, an encoder network learns to compress an input audio segment into a compact latent vector—the embedding itself—while a corresponding decoder network learns to reconstruct the original audio from only this vector. Other popular approaches learn embeddings through different objectives. Selfsupervised models like Wav2Vec 2.0 operate directly on raw waveforms, learning rich features by predicting masked or missing portions of the audio signal. Another common strategy involves transfer learning, where models like VGGish first convert audio into a spectrogram—a visual representation—and then process this image with a convolutional neural network originally designed for computer vision tasks [9]. Finally, multimodal models such as CLAP use contrastive learning to create a shared embedding space between audio and text, learning to align sounds with their corresponding descriptions [10]. 2.3.1 Contrastive Learning Contrastive learning is a machine learning paradigm in which a model to distinguish between similar and dissimilar data points. The core idea is to learn an 2.3. Audio embeddings 11 embedding space where "positive pairs" (similar items) are brought closer together, while "negative pairs" (dissimilar items) are pushed further apart. For any given data point, referred to as the "anchor," a positive sample is a related item (e.g., an augmented version of the same image), and negative samples are all other items in a training batch [8]. By optimizing a contrastive loss function, the model learns to produce representations that cluster semantically similar items without needing explicit, human-provided labels for every single class. OpenAI’s CLIP (Contrastive Language-Image Pre-training) model applies this principle to bridge the gap between vision and text. CLIP utilizes a dual-encoder architecture: one encoder processes images, and another processes text descriptions. During its pre-training phase, the model is fed a massive dataset of (image, text) pairs collected from the internet. For each pair, the image and its corresponding text form a positive pair, while the image and the text from all other pairs in the batch are treated as negative pairs. The model’s objective is to maximize the cosine similarity of the embeddings for the correct image-text pairs while minimizing it for all incorrect pairs. This process effectively aligns the two modalities into a single, shared embedding space [11]. This same contrastive methodology has been successfully extended to the audio domain with models like CLAP (Contrastive Language-Audio Pre-training). CLAP learns a joint embedding space that aligns sounds with their corresponding text descriptions. By training on pairs of audio clips and their textual metadata, CLAP can perform zero-shot audio classification, identifying sounds based on natural language queries (e.g., "a dog barking" or "a car horn"). This cross-modal capability is crucial for applications like audio retrieval, where users can search vast sound libraries using descriptive text rather than other audio examples [10]. Chapter 3 VAEs, GANs, and RAVE This chapter will introduce and explain Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and the RAVE model. 3.1 Variational Autoencoders Variational Autoencoders (VAEs) constitute a class of generative neural networks that transcend conventional autoencoder limitations by incorporating probabilistic modeling principles. Unlike traditional autoencoders that produce deterministic latent representations, VAEs encode input data as probability distributions within a continuous latent space, enabling the generation of novel data samples that maintain structural fidelity to the training distribution. This probabilistic framework, originally formulated by Kingma and Welling in 2013, revolutionized generative modeling by providing a mathematically principled approach to generation while somewhat preserving the interpretability and controllability of the learned representations [12]. VAEs are comprised of an encoder network that maps input observations to distributional parameters (mean and variance) in a latent space, a stochastic sampling mechanism facilitated by the reparameterization trick, and a decoder network that reconstructs data from sampled latent vectors. The model optimizes a composite objective function that balances reconstruction fidelity against regularization con12 3.1. Variational Autoencoders 13 straints, specifically minimizing the Kullback-Leibler divergence between the learned posterior distribution and a specified prior distribution. This dual optimization creates a structured latent manifold where semantically similar inputs cluster together, facilitating meaningful interpolation and controlled generation through latent space manipulation [12]. VAEs demonstrate exceptional utility in research applications requiring both data compression and generation capabilities, offering superior training stability compared to adversarial approaches while maintaining theoretical grounding in variational inference principles. The structured latent space facilitates rigorous analysis of model behavior while the generative capabilities enable comprehensive evaluation across diverse scenarios. 3.1.1 Marginal Likelihood The mathematical foundation of VAEs rests on the intractable marginal likelihood problem. Given observed data xand latent variables z, we seek to maximize the marginal likelihood pθ(x) = Rpθ(x|z)pθ(z)dz, where θrepresents the parameters of our generative model. Since this integral is computationally intractable for machine learning models, VAEs introduce an approximate posterior qϕ(z|x)parameterized by ϕ, typically implemented as a neural network encoder. The key insight lies in deriving a tractable lower bound on the log marginal likelihood through Jensen’s inequality, yielding the Evidence Lower Bound (ELBO): L(θ, ϕ;x) = Eqϕ(z|x)[log pθ(x|z)] −DKL(qϕ(z|x)∥pθ(z)) This formula decomposes the objective into two components: a reconstruction term that ensures the decoder can accurately reconstruct inputs from latent representations, and a regularization term that constrains the approximate posterior to align with the specified prior distribution [12]. 14 Chapter 3. VAEs, GANs, and RAVE 3.1.2 KL-Divergence The Kullback-Leibler divergence serves as a fundamental measure that quantifies the dissimilarity between two probability distributions, functioning as a crucial regularization mechanism in VAEs that prevents posterior collapse and ensures meaningful latent space structure [12]. Mathematically defined as DKL(P∥Q) = Ex∼P[log P(x)−log Q(x)] for continuous distributions, the KL divergence is asymmetric and always non-negative, reaching zero only when the distributions are identical. In the VAE framework, DKL(qϕ(z|x)∥p(z)) specifically measures how the learned approximate posterior deviates from the specified prior distribution, typically a standard multivariate Gaussian N(0,I). When both distributions are Gaussian, this divergence admits a closed-form solution: DKL =1 2PJ j=1(1 + log(σ2 j)− µ2 j−σ2 j), where Jrepresents the latent dimensionality [12]. This regularization term prevents the encoder from learning arbitrarily complex posterior distributions that would overfit to the training data, instead encouraging the latent space to maintain continuity and structure essential for interpolation and generation. The balance between reconstruction accuracy and KL regularization creates a natural trade-off that shapes the learned representations, with higher KL weights promoting more structured but potentially less expressive latent spaces, while lower weights risk posterior collapse where latent variables become uninformative. 3.1.3 Reparameterization Trick The critical innovation enabling end-to-end optimization lies in the reparameterization trick, which transforms the stochastic sampling operation into a differentiable function. Rather than directly sampling from qϕ(z|x), VAEs reparameterize the latent variable as z=µϕ(x) + σϕ(x)⊙ϵ , where ϵ∼ N (0,I)represents auxiliary noise independent of the model parameters, and ⊙denotes element-wise multiplication. This transformation allows gradients to flow through the sampling process, enabling standard backpropagation while main- 3.1. Variational Autoencoders 15 taining the stochastic nature essential for regularization and generation [12]. The mathematical elegance of this approach provides VAEs with superior training stability compared to adversarial methods while preserving the theoretical guarantees of variational inference, making them particularly suitable for research applications requiring both interpretable latent representations and reliable generative capabilities. Figure 1: A visual depiction of a typical VAE architecture. The encoder takes the input data and compresses it into a latent representation, typically via a series of convolutional layers. The decoder then decompresses this representation and seeks to produce a perceptually identical output. 3.1.4 Conditioning VAEs Conditional Variational Autoencoders (cVAEs) extend standard VAEs by introducing side information or labels (often denoted as y) into both the encoder and decoder networks, enabling controllable and structured generation. As described in a detailed in an academic overview of conditioning in VAEs [13], the conditioning variable can represent a wide range of semantic concepts such as class, style, or any structured language-based attribute you wish the model to learn and generate. There are two principal approaches to incorporating conditioning: assuming the latent variable zand the conditioning variable yare independent, or making them dependent. In the independent case, the prior for zand yfactorizes as p(z, y) = p(z)p(y), which 16 Chapter 3. VAEs, GANs, and RAVE promotes disentangled representations. This is desirable for tasks where you want z to capture "everything except" the information in y, making the generative model more controllable (e.g. Fader Networks) [13]. Alternatively, the dependent case sets a conditional prior p(z|y)p(y), often resulting in a more expressive model but with less independent control over each source of variation. The cVAE’s objective function encourages the learned posterior q(z|x, y)to approximate the specified (potentially conditional) prior, while the decoder p(x|z, y) reconstructs the data from both zand y. Tuning the regularization term (i.e., the weight on the KL divergence) is critical: if it’s too low, the latent variable may "leak" information about y, making the conditioning ineffective; if it’s too high, the model’s generation quality degrades due to the forced independence or matching to a too-simple prior. The discussion also highlights that the independent cVAE formulation is particularly suitable for controllable generation (like swapping semantic attributes), while the dependent variant is often more powerful for conditional data modeling. Ultimately, the chosen conditioning strategy depends on the desired trade-off between interpretability/controllability, and the quality of the generation [13]. 3.2 Generative Adversarial Networks Generative Adversarial Networks (GANs) introduce a method of deep learning that pits two neural networks against each other. The approach mimics an adversarial game between a counterfeiter and a detective, with one network generating synthetic data while the other tries to detect it [14]. The framework operates on a simple principle. The generator network creates synthetic samples that resemble real training data, and the discriminator network evaluates these samples and determines their authenticity. Through this competition, both networks improve their performance [14]. The generator becomes better at 3.2. Generative Adversarial Networks 17 creating realistic data, while the discriminator becomes more skilled at detection. This adversarial training process produces remarkable results. GANs can generate images, text, audio, and other data types that humans often cannot distinguish from real examples. The applications span numerous fields, from computer vision to natural language processing. The mathematical foundation builds on game theory. The generator and discriminator engage in a minmax game, where each network tries to optimize its objective function. The generator minimizes the discriminator’s ability to classify its outputs as fake. The discriminator maximizes its classification accuracy between real and generated samples. 3.2.1 Architecture The GAN architecture for audio generation employs the same fundamental adversarial principle as image generation but adapts to the unique challenges of temporal data. The generator network typically receives random noise as input and transforms it into coherent audio sequences, though this differs in the RAVE approach as will be discussed later. The discriminator architecture often mirrors the generator design but in reverse. Raw waveform discriminators use one-dimensional convolutions with stride to reduce temporal dimensions while extracting increasingly abstract features as data propagates through neural network layers. The network processes audio sequences through multiple layers, each detecting patterns at different time scales. After producing a compressed, high-level feature representation through this process, the final layer completes the classification task by producing a single probability that indicates whether the input represents real or generated audio. 3.2.2 Training During each training step, the two networks optimise separate but coupled loss functions that form a mini–max game 1. The discriminator loss is defined as: 24 Chapter 4. Experiment 4.1.1 Dataset To facilitate the training of a controllable generative system, a specialized dataset was created consisting of 4 hours of synthesized footstep audio clips. This dataset was purpose-built to provide clean, continuous, attribute labels essential for conceptaware vectors for generalizable attribute detection in footstep audio samples [15]. Each audio clip is associated with four specific floating-point values that define its perceptual characteristics: ‘speed‘, ‘grass‘, ‘wood‘, and ‘concrete‘. ’Grass’, ’wood’, and ’concrete’ refer to the degree to which each respective material is present on the ground upon which the footsteps land. For the purposes of this experiment, we passed two attributes to the model in the hopes of achieving disentanglement: "grassiness" and "speed". By using synthesized audio, we ensure that these attribute values represent a ground truth, allowing us to confidently train the latent discriminator without worrying about overly noisy attribute data. The dataset’s structure was systematically designed to cover a wide range of attribute combinations, allowing the model to observe how changes in each parameter independently affect the resulting sound. The continuous nature of the labels is particularly crucial, as it enables the use of regression-based techniques to identify a coefficient vector capable of producing an attribute score when combined with an audio embedding vector. With 2401 unique samples, the dataset provides sufficient density and variety to train a model that can robustly map these semantic concepts to specific, controllable dimensions in the latent space, forming the foundation for the "semantic fader" control system investigated in this experiment. 4.1.2 Embedding and Attribute Computation The core of this project’s approach to attribute control lies in translating abstract, human-understandable concepts like "speed" or "grassiness" into concrete mathematical directions within a model’s latent space. To achieve this, we compute concept vectors for each desired semantic attribute. This method is inspired by the foundational work on Concept Activation Vectors (CAV) by Kim et al [16]. While 4.1. Experiment design 25 their work originally used linear classifiers to distinguish the presence or absence of a concept for interpretability, this project adapts the core idea for a generative purpose. Instead of a binary classification, we treat the task as a continuous regression problem, which is better suited for the smoothly varying attributes in our synthesized audio dataset. The specific technique employed here is what we term a Regression Concept Vector (RCV). For each attribute (e.g., speed, grassiness), a separate linear regression model is trained. This model learns to predict the continuous ground-truth value of the attribute (e.g., a float from 0.0 to 1.0) directly from the high-dimensional latent space embeddings produced by the CLAP model. The key insight is that the learned coefficient vector of this linear regressor represents the direction in the latent space of the CLAP model that corresponds most strongly to an increase in that specific attribute’s value. This vector and its ability to predict semantic attributes provides the basis for the disentanglement of the downstream VAE’s latent representation. This regression-based approach for steering generative models aligns with a broader research effort into creating more disentangled and controllable representations. By framing concept-to-vector mapping as a regression task, we can directly leverage labeled data to create controls that are not just representative of whether or not an attribute is present in data but also exhibit to what degree the attribute is present. This allows for a more intuitive and powerful method of "steering" the generative process. This RCV is not used to manipulate the CLAP space directly. Instead it serves as a crucial tool for structuring the latent space of the core generative model. The RCV’s ability to produce a scalar attribute score from an audio embedding provides the quantitative basis for the adversarial disentanglement of the downstream VAE’s latent representation. In the main training loop, the latent discriminator is tasked with predicting these attribute scores from the VAE encoder’s output. The encoder, in turn, is trained to produce a latent code that "fools" the discriminator, thereby learning a representation that is explicitly invariant to—or disentangled from—the attribute in question. This allows the ground-truth attribute value to be appended 26 Chapter 4. Experiment later as a conditional input to the decoder, facilitating precise and independent control. 4.1.3 Audio Representation The choice of audio representation as input to the VAE model is crucial and has a great impact on the model’s outcomes. We decided to convert the raw WAV files to a 16-band Pseudo-Quadrature Mirror Filter (PQMF) bank, a specialized digital signal processing tool used to efficiently decompose a single, high-bandwidth signal into multiple, lower-bandwidth sub-band signals [17]. The core of the PQMF system is an analysis filter bank, which takes the input audio and passes it through a series of parallel band-pass filters. Each filter is designed to isolate a specific frequency range. After filtering, the signal in each sub-band is downsampled. Because each sub-band now contains a much narrower range of frequencies, its sampling rate can be significantly reduced without losing information, a process known as critical sampling. This decomposition is highly efficient as it reduces the temporal resolution and computational load required to process audio data in subsequent steps [17]. Figure 3: Overview of the PQMF architecture The main VAE model is tasked with taking in this PQMF representation and providing a faithful reconstruction. To get back to the original audio, the sub-band 4.1. Experiment design 27 signals are fed into a corresponding synthesis filter bank. First, each sub-band signal is upsampled by inserting zeros between the existing samples to return it to the original sampling rate. Then, each upsampled signal is passed through a synthesis filter that mirrors the properties of its corresponding analysis filter. The outputs from all the synthesis filters are then summed together to produce a replica of the original full-bandwidth signal. We chose this representation due to its effectiveness in making the computationally expensive task of processing raw audio more tractable by representing the audio as a collection of multiple frequency bands, and its ability to achieve a near-perfect reconstruction of the original signal [17]. Instead of forcing the VAE to learn the intricate structure of the full-bandwidth audio at once, the PQMF splits the signal into multiple lower-frequency sub-bands. This decomposition significantly reduces the temporal resolution that the encoder and decoder must process within each band, making the learning task computationally more efficient and stable. Furthermore, since the reconstruction loss is calculated directly on these perceptually relevant sub-bands, the model is optimized to minimize errors in distinct frequency ranges, which often leads to higher-fidelity audio synthesis. The near-perfect reconstruction property of the PQMF ensures that the audio quality is preserved during the encoding and decoding stages, allowing the model to focus entirely on the generative task without introducing artifacts. 4.1.4 Model Architecture The model is a conditional Variational Autoencoder (VAE) enhanced with an adversarial component. The core of the model consists of an encoder, which compresses the multi-band audio into a lower-dimensional latent space, and a conditional decoder, which reconstructs the audio from a sample of this latent space combined with explicit attribute information. The VAE’s architecture is defined by a symmetric encoder-decoder structure operating on a 16-band PQMF representation of the audio. The Encoder is a deep 28 Chapter 4. Experiment convolutional network that progressively downsamples the input through four main blocks with temporal downsampling ratios of [4, 4, 4, 2]. This compresses the temporal dimensionality of the input audio by a factor of 128. The decoder mirrors this architecture, using four corresponding upsampling blocks to reconstruct the 16-band audio from a latent vector. Importantly, the decoder is conditional: its input is a 130-dimensional vector created by concatenating the 128-dimensional latent code with a 2-dimensional attribute vector (speed and grassiness), enabling controlled synthesis. Several key hyperparameters govern the model’s training and behavior. The latent space has a dimensionality of 128, providing a compact representation of the audio features. The VAE loss function is balanced by a term weighting the KL divergence, which is set to 0.2 during the training step. The model is trained using the Adam optimizer with a learning rate of 1e-3 and a batch size of 16. Audio inputs are processed as 6-second clips sampled at 44.1 kHz. The training process uses a hybrid objective function that combines classic VAE losses with an adversarial loss. The VAE is trained to minimize a reconstruction loss, which is calculated as a multi-scale spectral distance between the original and generated audio sub-bands to ensure perceptual similarity. Alongside this, a Kullback-Leibler (KL) divergence loss regularizes the encoder, pushing the distribution of the latent space to approximate a standard gaussian prior distribution. This regularization is crucial for ensuring that the latent space is smooth and continuous, allowing for meaningful interpolation and generation of novel audio samples. Latent discriminator The adversarial component introduces a unique dynamic to the training. A ‘latent discriminator‘ is trained concurrently to predict the audio’s descriptive attributes directly from the latent code produced by the encoder. The discriminator is rewarded for making correct predictions and the encoder is rewarded for fooling the discriminator. This adversarial objective forces the encoder to learn a disentangled latent representation where specific dimensions correlate with the audio attributes. 4.1. Experiment design 29 The decoder then leverages this structured latent code along with the ground-truth attributes to perform conditional synthesis, enabling control over the characteristics of the generated audio. The architecture of the discriminator consists of two main parts: a shared feature extraction backbone followed by attribute-specific prediction heads. The input to the model is the latent code ’z’. The shared feature extraction stage begins with a series of blocks, each containing a 1d convolutional layer with a kernel size of 7, followed by batch normalization and a LeakyReLU activation. These initial layers process the latent sequence while maintaining its channel dimension. After these repeating blocks, an additional convolutional block reduces the channel dimension by half. This shared representation is then fed into a separate prediction head for each attribute. Each head further refines the features, first with a convolutional block that reduces channels to 32, and then with a final convolutional layer that acts as the classifier, mapping the 32-channel representation to an output with output channels equalling the number of quantization bins. This final output contains the raw logits for each quantized bin for every time step in the sequence. This discriminator approach mirrors the one used by Devis et al. in their 2023 paper [2]. Attribute computation and prediction The computed semantic attributes are stored as a vector of length equal to the temporal dimensionality of the input audio’s latent representation to allow for appending before being passed to the decoder. Before model training, quantization bins are computed on the entire dataset by mapping the set of continuous attribute values to a discrete set, enabling the discriminator to handle attribute computation as a classification problem. All attribute vectors are quantized into 16 bins to match the output of the latent discriminator. The accuracy of the discriminator is then assessed by cross-entropy loss. Quantization is the process of mapping a large, continuous set of values into a smaller, discrete set, often referred to as bins. In models like F-RAVE, which deal with continuous attribute prediction, quantization enables the model to handle at- 30 Chapter 4. Experiment tribute prediction as a classification problem by assigning continuous predictions to a finite number of bins. In this implementation, quantization was done by continuous feature values (scaling them to the range [−1, 1]), sorts these values, and then defining bin thresholds by selecting equidistant indices across the sorted data, effectively dividing the attribute range into 16 bins. The latent discriminator predicted attributes via logit computation across all 16 attribute bins. Cross-entropy loss was then computed between the real and predicted attribute values. Formally, cross-entropy loss is defined as L=− K X k=1 yklog(pk) where Kis the number of classes, ykis the true label, and pkis is the predicted probability for class k. Chapter 5 Results This chapter will exhibit the results of the experiment detailed in the previous chapter, beginning with the performance of the attribute computation method and moving to the perceptual results of modifying the latent axes corresponding to the disentangled semantic attributes. 5.1 Attribute Computation 5.1.1 Speed The RCV dedicated to speed (Fig. 4-1) achieves an R²of 0.990 on the training data and 0.984 on the unseen test clips, indicating that the learned regression coefficient is highly predictive of footstep tempo. 5.1.2 Surface Material Attributes While performance stays solid for the "grassiness" attribute (R²= 0.988), prediction quality drops markedly for the two other “material” dimensions: Upon perceptual evaluation of the dataset, this is likely due to a high level of entanglement between "grassiness" and the other material attributes. For example, adding "woodiness" to a clip with high "grassiness" resulted in much less of a perceptual difference 31 32 Chapter 5. Results than performing the inverse operation. Due to this entanglement, only "speed" and "grassiness" were chosen as perceptual attributes to disentangle during training. 5.1.3 Plots Figures 4-1 to 4-4 summarize the quality with which the learned regression-concept vectors (RCVs) recover the four continuous attributes encoded in the foot-step dataset. Each panel shows the predicted value against the ground-truth value for the test set; the dashed red line indicates perfect prediction. The coefficient of determination (R2) obtained from the full linear regressor are annotated in each plot (a) Speed (b) Woodiness (c) Grassiness (d) Concrete Figure 4: These four plots show test performance of the learned RCV for each of the considered semantic attributes in this experiment. The "speed" and "grassiness" RCVs perform well, while the other attributes are harder to predict. 5.2. Semantic Fader Model 33 5.2 Semantic Fader Model This section will detail the quantitative and perceptual results related to training and inference of the proposed VAE/GAN generative audio model on the synthesized footsteps dataset. Latent Size Ratios N PQMF Bands Num Bins 128 [4, 4, 4, 2] 16 16 Table 1: Model configuration of the most successful training run. "Latent Size" refers to the number of dimensions present in the model’s latent representation. "Ratios" refers to the downsampling/upsampling ratio at each convolutional layer of the encoder/decoder. The product of these ratios gives the total sampling factor = 128. "N PQMF Bands" refers to the amount of frequency sub-bands the audio was decomposed into as input to the model. "Num Bins" refers to the amount of bins used to quantize the continuous attributes. 5.2.1 Reconstruction Fidelity Reconstruction fidelity was measured via a multi-scale spectral distance computed on PQMF sub-bands. Given the multiband signals xmb, ymb ∈RB×C×T,we apply magnitude STFTs at scales S={2048,1024,512,256,128}with hop s/4to each band and aggregate across bands and scales. At each scale s, the distance couples a relative L2 term on linear magnitudes with an L1 term on log-magnitudes (stabilized by ε= 10−7), encouraging both envelope matching and contrast at different loudness levels. The formal definition is given below: Lrecon(x, y) = X s∈S ∥|Xs|−|Ys|∥2 ∥|Xs|∥2 + log(|Xs|+ε)−log(|Ys|+ε) 1!, where Xs= STFTs(xmb)and Ys= STFTs(ymb). The loss is computed per PQMF band (by reshaping bands into the batch) and summed over scales, yielding the scalar spectral_distance used for training loss. The reconstruction loss curve in figure 5 exhibits a rapid initial decrease followed by a slower, steady improvement, converging near a stable regime after roughly the mid-training stage. Listening tests indicated strong preservation of overall "shape" and amplitude envelope, but comparatively less high-frequency and fine-grained 40 Chapter 6. Discussion 6.3 Future Work Challenges faced in achieving temporally consistent and coherent outputs indicate that a different strategy may need to be applied to time-dependent semantic features. In particular, deep-learning architectures that better account for temporal sequences such as RNNs, LSTMs, and Transformers may be better suited for this aspect. A comprehensive solution may come in the form of a hybrid VAE-RNN architecture that combines the representational power of statistical distribution modeling with the sequence modeling capabilities of recurrent neural networks. List of Figures 1 A visual depiction of a typical VAE architecture. The encoder takes the input data and compresses it into a latent representation, typically via a series of convolutional layers. The decoder then decompresses this representation and seeks to produce a perceptually identical output. 15 2 A visual depiction of a typical GAN architecture. The generator creates fake data from a compressed latent representation, and the discriminator performs the reverse process (downsampling) in order to undertand and discern between real and fake data. . . . . . . . . . 19 3 Overview of the PQMF architecture . . . . . . . . . . . . . . . . . . . 26 4 These four plots show test performance of the learned RCV for each of the considered semantic attributes in this experiment. The "speed" and "grassiness" RCVs perform well, while the other attributes are hardertopredict.............................. 32 5 Reconstruction loss over roughly 750,000 training steps . . . . . . . . 34 6 Computed speed for original and auto-encoded audio clips . . . . . . 35 7 Computed grassiness for original and auto-encoded audio clips . . . . 35 8 Loss curve during initial training with GAN reading latent codes. . . 38 41 List of Tables 1 Model configuration of the most successful training run. "Latent Size" refers to the number of dimensions present in the model’s latent representation. "Ratios" refers to the downsampling/upsampling ratio at each convolutional layer of the encoder/decoder. The product of these ratios gives the total sampling factor = 128. "N PQMF Bands" refers to the amount of frequency sub-bands the audio was decomposed into as input to the model. "Num Bins" refers to the amount of bins used to quantize the continuous attributes. . . . . . . . . . . . 33 2 Parameters related to discriminator loss weight control. "Initial Lambda" refers to the initial value of the loss term weight. "Lambda Delay" refers to the amount of steps before the ramp up began. "Max Lambda" refers to the maximum weight term that was applied after thelinearrampup. ........................... 38 42 Bibliography [1] Lample, G. et al. Fader networks: Manipulating images by sliding attributes (2018). URL http://arxiv.org/abs/1706.00409. [2] Devis, N., Demerlé, N., Nabi, S., Genova, D. & Esling, P. Continuous descriptor-based control for deep audio synthesis (2023). URL http://arxiv. org/abs/2302.13542. [3] Caillon, A. & Esling, P. Rave: A variational autoencoder for fast and highquality neural audio synthesis (2021). URL http://arxiv.org/abs/2111. 05011. [4] Burgess, C. P. et al. Understanding disentangling in -vae (2018). URL http: //arxiv.org/abs/1804.03599. [5] Yang, X., Bi, W., Sun, Y., Cheng, Y. & Yan, J. Towards better understanding of disentangled representations via mutual information (2020). URL http: //arxiv.org/abs/1911.10922. [6] Kim, H. & Mnih, A. Disentangling by factorising (2019). URL http://arxiv. org/abs/1802.05983. [7] van den Oord, A., Vinyals, O. & Kavukcuoglu, K. Neural discrete representation learning (2018). URL http://arxiv.org/abs/1711.00937. [8] van den Oord, A. et al. Wavenet: A generative model for raw audio (2016). URL http://arxiv.org/abs/1609.03499. 43 44 BIBLIOGRAPHY [9] Hershey, S. et al. Cnn architectures for large-scale audio classification (2017). URL http://arxiv.org/abs/1609.09430. [10] Wu, Y. et al. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation (2024). URL http://arxiv.org/ abs/2211.06687. [11] Radford, A. et al. Learning transferable visual models from natural language supervision (2021). URL http://arxiv.org/abs/2103.00020. [12] Kingma, D. P. & Welling, M. An introduction to variational autoencoders (2019). URL http://arxiv.org/abs/1906.02691http://dx.doi.org/10. 1561/2200000056. [13] Beckham, C. A deep dive into conditional variational autoencoders. https: //beckham.nz/2023/04/27/conditional-vaes.html. [14] Goodfellow, I. J. et al. Generative adversarial networks (2014). URL http: //arxiv.org/abs/1406.2661. [15] Wyse, L. & Kellock, P. Embedding interactive sounds in multimedia applications. Tech. Rep. (1999). [16] Kim, B. et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav) (2018). URL http://arxiv.org/abs/ 1711.11279. [17] Mimilakis, S. I. & Schuller, G. Investigating the potential of pseudo quadrature mirror filter-banks in music source separation tasks (2017). URL http: //arxiv.org/abs/1706.04924. The full repository containing the code written for this project can be found at https://github.com/JedPadoa/SemanticFader