Comparison of Audio Encoders for Audio-Text Contrastive Learning Representations

Master thesis on Sound and Music Computing Universitat Pompeu Fabra Comparison of Audio Encoders for Audio-Text Contrastive Learning Representations Sergio Cárdenas Gracia Supervisor: Pablo Alonso Jiménez Co-Supervisor: Dmitry Bogdanov July 2025 Master thesis on Sound and Music Computing Universitat Pompeu Fabra Comparison of Audio Encoders for Audio-Text Contrastive Learning Representations Sergio Cárdenas Gracia Supervisor: Pablo Alonso Jiménez Co-Supervisor: Dmitry Bogdanov July 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 1 1.2 Scopeoftheproject ............................ 2 1.3 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 State of the Art 3 2.1 Contrastivelearning ............................ 3 2.2 Textencoders................................ 4 2.3 Audioencoders ............................... 4 2.4 Modeltraining ............................... 5 2.5 Trainingdatasets.............................. 6 3 Methodology 7 3.1 Trainingsetup................................ 7 3.1.1 Dataset ................................... 7 3.1.2 Modelarchitecture ............................. 7 3.2 Evaluationsetup .............................. 8 3.2.1 Zero-shot classification on the GTZAN dataset . . . . . . . . . . . . . . 8 3.2.2 Multi-label classification on the MagnaTagATune dataset . . . . . . . . 9 3.2.3 Text-to-music retrieval on the Song Describer dataset . . . . . . . . . . 9 3.3 Experiments................................. 10 3.3.1 HTSAT-base (initialized weights) + RoBERTa (frozen) . . . . . . . . . 10 3.3.2 MAEST-10s (initialized weights) + RoBERTa (frozen) . . . . . . . . . 11 3.3.3 Hyperparameters exploration . . . . . . . . . . . . . . . . . . . . . . . 11 3.3.4 Evaluationmethods............................. 11 4 Results 12 4.1 HTSATvsMAEST............................. 12 4.2 Hyperparameters effect . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.2.1 Batchsize.................................. 13 4.2.2 WeightDecay................................ 15 4.2.3 LearningRateDecay............................ 16 4.3 Evaluationmethods............................. 16 5 Conclusions 18 5.1 Discussion.................................. 18 5.2 Conclusions ................................. 19 5.3 Futurework................................. 20 List of Figures 21 List of Tables 22 Bibliography 23 A Training and Validation Loss Graphs 26 Acknowledgement I would like to express my sincere gratitude to my supervisors, Pablo Alonso and Dmitry Bogdanov, for their continuous support and invaluable guidance throughout this project. Their expertise and ambition have been essential to my learning and progress. I am also deeply thankful to my family and friends for their unwavering support and encouragement, which have been a constant source of strength. Finally, I would like to thank my colleagues from the Master’s in Sound and Music Computing. Sharing this journey with them has been a rewarding experience, filled with many memorable moments and mutual learning. 4Chapter 2. State of the Art While this fusion strategy shows promise for general audio, it performs poorly in the context of music. As a result, the fusion-based LAION-AI CLAP is not considered suitable for this project. Instead, the non-fusion variant of LAION-AI CLAP is chosen as the project’s baseline, as it offers a more recent and refined implementation compared to the original Microsoft CLAP. 2.2 Text encoders Text encoders are essential for generating meaningful representations of textual data that can be matched with audio content. Current state-of-the-art text encoders for multimodal tasks are typically based on transformer architectures [4]. While OpenAI’s CLIP used a transformer-based text encoder trained from scratch, recent best practices in multimodal learning favor using pretrained language models such as BERT [5] or RoBERTa [6]. Microsoft CLAP adopts BERT as its text encoder, whereas LAION-AI CLAP uses RoBERTa. Although other pretrained encoders have been explored in multimodal contexts, BERT and RoBERTa remain the most widely adopted for audio–text alignment, and the discussion here is therefore centered on them. Both produce high-quality, context-aware embeddings that can be aligned with audio representations. However, RoBERTa is considered a more optimized and robust variant of BERT, making it the preferred choice for current multimodal applications. 2.3 Audio encoders Audio encoders are critical for extracting meaningful features from raw audio that can be aligned with textual descriptions. In recent years, PANN [7] and HTSAT [8] have been widely selected as the primary audio encoders for multimodal tasks in the audio-text domain. PANN (Pre-trained Audio Neural Network) is a CNN-based audio classification model with 7 downsampling CNN blocks and 7 upsampling blocks. HTSAT (Hierarchical Token-Semantic 2.4. Model training 5 Audio Transformer), on the other hand, is a transformer-based model that uses four groups of Swin Transformer blocks [9] to capture complex patterns in audio spectrograms with a focus on optimization. These encoders were used in Microsoft CLAP and LAION-AI CLAP, respectively (PANN in the former and HTSAT in the latter). Among the two, HTSAT is generally considered the more suitable option, particularly due to its strong performance on a variety of audio understanding tasks. That said, MAEST [10] (Music Audio Efficient Spectrogram Transformer) is a newer encoder specifically designed for music-related tasks. Compared to HTSAT, it is a larger and more complex model, and it has shown promising results. This makes it a valuable candidate to evaluate in a contrastive learning setup focused on music, particularly in low-data scenarios where model efficiency and generalization are critical. 2.4 Model training Training a contrastive learning model involves minimizing a loss function that brings matching audio-text pairs closer together in the embedding space, while pushing apart non-matching pairs. To achieve this, both audio and text inputs are passed through their respective encoders and then projected into a shared embedding space using modality-specific linear projections. Although training all components jointly is standard practice, this approach can be computationally expensive. To address this, several alternatives have been proposed [11, 12], which aim to reduce training costs without significantly sacrificing performance. One practical strategy in multimodal settings is to initialize the audio encoder with pretrained weights from a related domain, such as vision or general audio classification, rather than training from scratch. Another common approach is to freeze the text encoder, especially when it has already been pretrained on large-scale language corpora. These techniques help reduce computational demands while still enabling 6Chapter 2. State of the Art effective contrastive learning. 2.5 Training datasets Training contrastive learning models requires paired audio-text data, where the text can consist of tags or natural language descriptions. Typically, large-scale datasets are used to achieve strong performance in this setting. For reference, the LAION-AI CLAP model for music was trained on the LAIONAudio-630k dataset (630,000 audios of human activities, natural sounds and audio effects), the AudioSet (approximately 2 million samples of human-labeled 10second sound clips drawn from YouTube videos), and a combination of music-related datasets1. However, these datasets are either not specifically curated for music or contain noisy and low-quality annotations. Additionally, working with such large-scale datasets is not always practical due to computational limitations, especially in smaller-scale or academic environments. 1Check reference for more details on the music datasets: https://github.com/LAION-AI/ audio-dataset/blob/main/data_collection/README.md Chapter 3 Methodology 3.1 Training setup Establishing a solid training setup involves carefully selecting both the data and the model architecture. As discussed in earlier sections, this project prioritizes highquality, openly available audio-text datasets. For this purpose, the MTG-Jamendo dataset [13] is a strong fit. 3.1.1 Dataset For training, the MTG-Jamendo split-0 is used, which is a standard partition for research. This split consists of 32,859 audio-text pairs for training and 11,101 for validation. The dataset is converted into the WebDataset format, a structure optimized for scalable machine learning workflows. In this format, each pair is represented by an audio path and a comma-separated list of tags associated with the track. 3.1.2 Model architecture The models follow the LAION-AI CLAP architecture, which serves as the baseline for this project. This includes both the component structure and the training configuration. The architecture consists of separate encoders for audio and text, followed by linear projections into a shared embedding space. 7 8Chapter 3. Methodology To reduce training complexity, the text encoder is frozen during training, since it is already pretrained on large-scale language data. For the audio encoder, pretrained weights are used to avoid training from scratch. By default, the HTSAT is used as the audio encoder. However, the MAEST encoder is also integrated into the implementation to enable comparison. This makes it possible to train models using either encoder within the same framework. 3.2 Evaluation setup Choosing a meaningful evaluation strategy is essential for accurately assessing and comparing model performance. To this end, three distinct evaluation tasks are defined, each highlighting different aspects of multimodal learning: zero-shot classification using the GTZAN dataset [14], multi-label classification using the MagnaTagATune dataset [15], and text-to-music retrieval using the Song Describer dataset [16]. 3.2.1 Zero-shot classification on the GTZAN dataset Zero-shot classification tests a model’s ability to assign audio samples to predefined categories without task-specific training. Instead, the model matches audio to text representations of each category by comparing embeddings in a shared space. The GTZAN dataset is well-suited for zero-shot classification, as it provides 30second audio samples for 10 distinct music genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock. Each genre includes 100 samples, except for jazz, which has 99 due to a corrupted file. To perform zero-shot classification, a text embedding is created for each genre using a simple prompt format: “This is a {genre} song.” These serve as the class representations in the embedding space. Next, audio embeddings are computed for all tracks. For each audio embedding, similarity scores are calculated with each of the text embeddings using the dot product. The genre corresponding to the highest similarity score is selected as the predicted label. 3.2. Evaluation setup 9 Finally, predicted labels are compared to ground truth annotations, and the overall classification accuracy is then used to evaluate model performance on this zero-shot task. 3.2.2 Multi-label classification on the MagnaTagATune dataset Multi-label classification evaluates a model’s ability to assign multiple relevant tags to a single audio track. The MagnaTagATune dataset is well-suited for this task, containing 29-second clips annotated with one or more labels drawn from a pool of 188 music-related tags. To simplify the task and ensure reliable evaluation, only the 50 most frequent tags are considered, following a common practice. The dataset is divided into training (15,244 samples), validation (1,529 samples), and test (4,332 samples) subsets. Audio embeddings are precomputed for all tracks. A lightweight classifier is trained on top of these embeddings using a transfer learning approach. The classifier is a two-layer feedforward neural network (MLP) that takes an audio embedding as input and outputs probabilities for each of the 50 tags. A sigmoid activation function allows each tag to be predicted independently. Tags are assigned when their predicted probability exceeds a defined threshold, enabling multi-label predictions per track. Evaluation is conducted using the Area Under the Receiver Operating Characteristic Curve (AUROC), which measures the model’s capacity to distinguish between classes across various decision thresholds, and the Mean Average Precision (MAP), which evaluates the ranking quality and precision across all relevant labels, providing a robust framework for assessing how well the multimodal model captures diverse musical attributes in a multi-label context. 3.2.3 Text-to-music retrieval on the Song Describer dataset Text-to-music retrieval evaluates the model’s ability to retrieve relevant audio tracks based on natural language queries. The Song Describer dataset is well-suited for this 10 Chapter 3. Methodology task, as it contains audio tracks paired with human-written captions that describe musical content. For evaluation, the validated subset of the dataset is used, which includes 746 unique audio tracks. However, the dataset contains a total of 1,106 audio-text pairs, since some tracks are associated with multiple captions. This adds value from an evaluation perspective, as it allows the model to retrieve a correct audio track based on a caption that may not be its exact pair. To carry out the retrieval, embeddings are computed for all audio tracks and text captions. A similarity matrix is then built using the dot product between each text and audio embedding. Each caption is treated as a query, and all tracks are ranked based on similarity scores. Performance is evaluated using Median Rank (MedR), which measures the median position of the correct audio track across all queries, and Recall at K (R@k), which indicates the proportion of queries for which the correct track appears in the top k results. These metrics together provide a strong indication of how effectively the model aligns text descriptions with musical content. 3.3 Experiments This section outlines the experiments conducted throughout the project. 3.3.1 HTSAT-base (initialized weights) + RoBERTa (frozen) A CLAP model is trained using the HTSAT audio encoder with initialized pretrained weights2, alongside a frozen RoBERTa text encoder. The specific HTSAT implementation used is the base version, designed to process 10-second audio clips. This model serves as the baseline for the project. Training a model with the same encoders as LAION-AI CLAP, but on a smaller dataset, ensures a fair comparison with other experimental models. 2Pretrained weights are available at the following link: https://github.com/LAION-AI/CLAP/ blob/main/README.md#reproducibility 3.3. Experiments 11 3.3.2 MAEST-10s (initialized weights) + RoBERTa (frozen) An experimental CLAP model is trained using the MAEST audio encoder, specifically the “discogs-maest-10s-pw-129e” variant, which also processes 10-second audio segments. As with the baseline, the RoBERTa text encoder remains frozen, and pretrained weights initialize the audio encoder. The goal of this experiment is to compare the performance of the MAEST encoder against the baseline HTSAT within a multimodal representation setting. 3.3.3 Hyperparameters exploration To better understand the impact of training configurations, various hyperparameter settings are explored throughout the training process. A key parameter under investigation is batch size, which is known to be directly related to model performance in contrastive learning settings. However, due to computational constraints, the batch sizes used in this project are smaller than those commonly used in large-scale contrastive learning setups. Furthermore, several hyperparameters intended to enhance generalization in lowdata contexts are examined. In particular, weight decay and learning rate decay are explored as strategies to reduce overfitting and promote more stable training under low-data conditions. 3.3.4 Evaluation methods For evaluation, two approaches are considered for computing audio embeddings: one involves extracting embeddings from a randomly selected 10-second segment of each audio track, and the other computes embeddings for every 10-second segment and averages them to obtain the final representation. These methods help assess how the choice of segment affects the quality of audio representations. Chapter 4 Results 4.1 HTSAT vs MAEST The first comparison between models using different audio encoders has been conducted, with both models trained using a batch size of 64 and the default hyperparameters provided by the LAION-AI CLAP implementation. The results of this evaluation are shown in Table 1. Task Metric HTSAT MAEST Zero-shot classification (GTZAN) Accuracy 51.05 28.73 Multi-label classification (MTT) AUROC 0.807 0.786 MAP 0.284 0.263 Text-to-music retrieval (SD) MedR ↓140 198 R@1 0.94 0.80 R@5 4.42 2.95 R@10 9.92 5.36 Table 1: Comparison of performance for models trained using HTSAT and MAEST audio encoders with batch size = 64, both initialized with pretrained weights. The model using the HTSAT audio encoder demonstrates solid performance across the various tasks, particularly given the limited amount of training data used compared to the large-scale datasets employed in the original LAION-AI models. In 12 4.2. Hyperparameters effect 13 contrast, the model using the MAEST audio encoder shows significantly lower performance in most evaluation tasks. Notably, the MAEST-based model appears to suffer from overfitting, suggested by the validation loss increasing significantly while the training loss decreases, as shown in Figure 1 and Figure 2. To further explore this issue, the following section analyzes the impact of various hyperparameters, including those that may help improve generalization, such as weight decay and learning rate decay. 4.2 Hyperparameters effect This section explores the effects of three key hyperparameters: batch size, weight decay, and learning rate decay. Batch size is evaluated using the HTSAT baseline model to understand its impact on performance in contrastive learning tasks. In contrast, weight decay, which helps prevent overfitting by discouraging large weights, and learning rate decay, which reduces the learning rate across layers to enable more stable convergence, are explored as strategies to address the overfitting observed in the MAEST-based model. 4.2.1 Batch size The performance of the LAION-AI CLAP implementation, using the HTSAT-base audio encoder (initialized with pretrained weights) and the RoBERTa text encoder, has been evaluated with three different batch sizes: 16, 32, and 64. The results of this comparison are shown in Table 2. As shown in Table 2, the relationship between batch size and performance is inconsistent across tasks when using pretrained weights. While larger batch sizes show improvements in multi-label classification, performance in zero-shot classification and text-to-music retrieval is less consistent. This behaviour may result from the pretrained weights already providing structured semantic representations. As a result, the model becomes less sensitive to the number of negative samples per batch, and thus to batch size. 20 Chapter 5. Conclusions pretrained weights and a frozen RoBERTa text encoder proved to be the most effective configuration for the LAION-AI CLAP framework under limited resources. Another key insight is the critical role of data volume, not only for learning robust multimodal representations but also for achieving reliable and meaningful evaluation. Models trained with limited data remain substantially behind state-of-the-art performance, highlighting the dependence of contrastive learning methods on large datasets. Finally, under constrained resources that restrict batch size, performance appears more influenced by factors such as audio encoder weight initialization. The inability to train with larger batch sizes is a significant limitation of this study. 5.3 Future work A natural next step for this project is to investigate a simplified version of the MAEST-based model, for example, by removing certain layers or reducing its dimensionality, to improve its suitability for low-data settings. Another potential direction for future work involves training models under lowdata conditions but with larger batch sizes, which requires more computational resources. Additionally, expanding the amount of training data would enable a more comprehensive analysis of how both batch size and data volume influence model performance. Finally, a more exploratory path could involve experimenting with alternative encoder combinations, including newer audio and text encoders, or investigating different approaches to learning multimodal audio-text representations beyond the conventional contrastive learning framework, such as approaches benefiting from large language models (LLMs) [17]. List of Figures 1 Training loss curves: (a) audio encoders comparison, (b) batch size comparison, (c) weight decay comparison, (d) learning rate decay comparison. ................................ 26 2 Validation loss curves: (a) audio encoders comparison, (b) batch size comparison, (c) weight decay comparison, (d) learning rate decay comparison. ................................ 27 21 List of Tables 1 Comparison of performance for models trained using HTSAT and MAEST audio encoders with batch size = 64, both initialized with pretrainedweights. ............................ 12 2 Comparison of batch size performance for trained HTSAT models initialized with pretrained weights. . . . . . . . . . . . . . . . . . . . 14 3 Comparison of batch size performance for trained HTSAT models initialized with random weights. . . . . . . . . . . . . . . . . . . . . . 14 4 Comparison of weight decay effect for trained MAEST models initialized with pretrained weights. . . . . . . . . . . . . . . . . . . . . . . . 15 5 Comparison of learning rate decay effect for trained MAEST models initialized with pretrained weights. . . . . . . . . . . . . . . . . . . . 16 6 Comparison of evaluation methods for the LAION-AI CLAP pretrainedcheckpoint. ............................ 17 22 Bibliography [1] Elizalde, B., Deshmukh, S., Al Ismail, M. & Wang, H. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1–5 (IEEE, 2023). [2] Wu, Y. et al. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1–5 (IEEE, 2023). [3] Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748–8763 (PMLR, 2021). [4] Vaswani, A. et al. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017). [5] Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186 (2019). [6] Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). 23 24 BIBLIOGRAPHY [7] Kong, Q. et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 2880–2894 (2020). [8] Chen, K. et al. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 646–650 (IEEE, 2022). [9] Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10012–10022 (2021). [10] Alonso-Jiménez, P., Serra, X. & Bogdanov, D. Efficient supervised training of audio transformers for music representation learning. arXiv preprint arXiv:2309.16418 (2023). [11] Maniparambil, M. et al. Harnessing frozen unimodal encoders for flexible multimodal alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, 29847–29857 (2025). [12] Qin, J., Liu, C., Cheng, S., Guo, Y. & Arcucci, R. Freeze the backbones: a parameter-efficient contrastive approach to robust medical vision-language pretraining. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1686–1690 (IEEE, 2024). [13] Bogdanov, D., Won, M., Tovstogan, P., Porter, A. & Serra, X. The mtgjamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML), 1–3 (2019). [14] Tzanetakis, G. & Cook, P. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing 10, 293–302 (2002). [15] Law, E., West, K., Mandel, M. I., Bay, M. & Downie, J. S. Evaluation of algorithms using games: The case of music tagging. In ISMIR, 387–392 (2009). BIBLIOGRAPHY 25 [16] Manco, I. et al. The song describer dataset: A corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311.10057 (2023). [17] Gardner, J., Durand, S., Stoller, D. & Bittner, R. M. Llark: A multimodal instruction-following language model for music. arXiv preprint arXiv:2310.07160 (2023). Appendix A Training and Validation Loss Graphs (a) audio encoders comparison (b) batch size comparison (c) weight decay comparison (d) learning rate decay comparison Figure 1: Training loss curves: (a) audio encoders comparison, (b) batch size comparison, (c) weight decay comparison, (d) learning rate decay comparison. 26 27 (a) audio encoders comparison (b) batch size comparison (c) weight decay comparison (d) learning rate decay comparison Figure 2: Validation loss curves: (a) audio encoders comparison, (b) batch size comparison, (c) weight decay comparison, (d) learning rate decay comparison.