Research and Evaluation of Automatic Sound FX Classification in Freesound using the Universal Category System

Master in Sound and Music Computing Universitat Pompeu Fabra Research and Evaluation of Automatic Sound FX Classification in Freesound using the Universal Category System Madhav Jaideep Supervisor: Frederic Font July 2025 Master in Sound and Music Computing Universitat Pompeu Fabra Research and Evaluation of Automatic Sound FX Classification in Freesound using the Universal Category System Madhav Jaideep Supervisor: Frederic Font July 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 2 1.2 KeyConcepts................................ 5 1.2.1 Automatic Sound Classification (ASC) . . . . . . . . . . . . . . . . . . 5 1.2.2 Semantics and Acoustic Features . . . . . . . . . . . . . . . . . . . . . 5 1.2.3 Multimodal learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.4 Freesound.................................. 7 1.2.5 Universal Category System (UCS) . . . . . . . . . . . . . . . . . . . . . 7 1.2.6 Evaluation metrics and Performance Assessment . . . . . . . . . . . . . 7 1.3 Objectives.................................. 10 1.4 Structure of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2 State of the Art 12 2.1 History of Sound Classification . . . . . . . . . . . . . . . . . . . . . . 12 2.2 CurrentApproaches ............................ 13 2.2.1 Machine learning based approaches . . . . . . . . . . . . . . . . . . . . 13 2.2.2 Deep learning based approaches . . . . . . . . . . . . . . . . . . . . . . 15 2.3 Datasets, Sound FX libraries and Taxonomy . . . . . . . . . . . . . . 17 2.4 Universal Category System . . . . . . . . . . . . . . . . . . . . . . . . 19 2.5 UCSdeployment .............................. 20 3 Methodology 22 3.1 Dataset ................................... 22 3.1.1 PSE150K .................................. 22 3.1.2 FreesoundData............................... 23 3.1.3 Otherdatasets ............................... 27 3.2 FeatureExtraction ............................. 27 3.3 Classification with dataset subset (PSE8K) . . . . . . . . . . . . . . . . 29 3.3.1 Experimental Design and Model Implementation . . . . . . . . . . . . . 29 3.4 Large-Scale classification using PSE150K . . . . . . . . . . . . . . . . . 31 3.5 Evaluation on Freesound Data . . . . . . . . . . . . . . . . . . . . . . . 32 3.6 Cross-Domain Classification Analysis . . . . . . . . . . . . . . . . . . . 33 3.6.1 Embedding Space Visualization . . . . . . . . . . . . . . . . . . . . . . 33 3.6.2 Cross-ModalSearch............................. 33 3.7 Fine-Tuning using Transfer learning . . . . . . . . . . . . . . . . . . . . 34 4 Results 35 4.1 Rapid Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.1 Audio-Only Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.1.2 Text-Only Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.1.3 Multimodal Concatenation . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.1.4 Multimodal Weighted Fusion (alpha=0.8) . . . . . . . . . . . . . . . . 37 4.1.5 Model Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . 37 4.2 Evaluation of full dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.3 Evaluation of PSE trained models on Freesound Data . . . . . . . . . . 42 4.3.1 Results on Preliminary Freesound Dataset . . . . . . . . . . . . . . . . 42 4.3.2 Results on Extended Freesound Dataset . . . . . . . . . . . . . . . . . 42 4.4 Fine-tuningResults............................. 45 5 Discussion 47 5.1 Rapid Evaluation on Dataset Subset . . . . . . . . . . . . . . . . . . . 47 5.2 Training and Evaluation on the full dataset . . . . . . . . . . . . . . . 49 5.3 Evaluation on Freesound Data . . . . . . . . . . . . . . . . . . . . . . . 50 5.3.1 Preliminary Dataset Evaluation . . . . . . . . . . . . . . . . . . . . . . 50 5.4 Comprehensive Dataset Evaluation . . . . . . . . . . . . . . . . . . . . 51 5.4.1 t-SNE Analysis of Embedding Spaces Across Domains . . . . . . . . . . 52 5.4.2 Category Analysis: Performance Across Sound Classes . . . . . . . . . 54 5.5 Fine-Tuning Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.6 FutureWork................................. 57 6 Conclusion 59 List of Figures 60 List of Tables 61 Bibliography 62 Acknowledgement I would like to express my sincere gratitude to my supervisor, Frederic Font, for his continuous guidance, and insights throughout the course of this thesis. I am grateful for the interest taken in my work and helping me understand everything better as I progressed. I would also like to thank Panagiota Anastasopoulou for kindly reviewing my work and providing valuable insights, which greatly helped improve the quality of this thesis and guided me towards new directions. I would also like to thank Xavier Serra for his academic guidance throughout the period of my Master’s study. I am also grateful to all my faculties and the entire Music Technology Group at Universitat Pompeu Fabra for providing an amazing environment for learning and growing, and for their generous support and resources provided. I am also extremely thankful to all my peers and classmates for their encouragement, and for creating a supportive environment that made this journey both rewarding and enjoyable. Finally, I would like to express my heartfelt gratitude to my family in India—my parents, my sister, and my partner Divya—for their unwavering love and support throughout my journey as a Master’s student in Barcelona, without which this would not have been possible. 4Chapter 1. Introduction •Domain gap: The most critical issue is the significant domain mismatch between training data and real-world samples. A model trained on professionally recorded, clean audio is likely to misclassify or fail entirely when exposed to low-quality, noisy, or acoustically varying recordings from everyday settings. •Metadata Variability: Textual information of the sounds such as titles, tags, and descriptions of the sound can enhance classification through multimodal models[6]. However, in community-driven platforms, this metadata is usually inconsistent or sometimes even incorrect. This metadata variability weakens the model’s ability to leverage text alongside audio. •Label Subjectivity: Human perception of sounds is very context dependent and subjective. "What one person recognizes as ’clapping,’ another might label as ’applause’ or ’hand percussion’ or what one person tags as "storm", another might tag as "rain". This ambiguity in labeling creates label noise in training data and leads to confusion in model predictions. At the same time, It also raises a philosophical question about the "correctness" of a tag in the absence of a ground truth. •Class Imbalance: In many user-generated datasets, certain sound categories dominate while others are rarer. For example, everyday sounds like footsteps, city noise, nature sounds or phone notifications may appear frequently, while niche or more context-specific sounds are usually lesser in number. This may lead to unbalanced learning, where classification systems may favor overrepresented classes and neglect smaller ones. •Scalability: Beyond research accuracy, there’s also the motivation of building tools that are functionally reliable in professional environments. Sound designers, music producers, and developers increasingly rely on accurately searchable sound libraries, tagging systems, and audio retrieval tools. For these systems to be genuinely useful in real-world applications, they must function effectively across even with unstructured and unpredictable datasets. These converge on a key research question - can sound classification systems de- 1.2. Key Concepts 5 signed around professional datasets and taxonomies like UCS be adapted to work effectively on noisy and highly variable user-generated sound libraries? By investigating this problem, this thesis aims to contribute towards better understanding of sound classification and also to a broader goal of making intelligent automatic sound classification systems that are deployable across diverse environments. 1.2 Key Concepts 1.2.1 Automatic Sound Classification (ASC) Automatic Sound Classification refers to the use of computational algorithms to manage, classify, and retrieve sound effects by automatically assigning meaningful labels to audio recordings, mimicking the human ability to recognize and categorize sounds. Unlike humans, who classify sounds intuitively through context and experience, machines depend on large, annotated datasets created by human experts. These labeled recordings form the foundation for training models using signal processing, machine learning, or deep learning techniques. Through this process, systems learn to detect patterns in audio and perform accurate classifications. Typically, the workflow begins with pre-processing, where raw recordings are cleaned, standardized, and segmented. Next, features are extracted from the waveform in a structured format suitable for computation. Finally, machine learning or deep learning algorithms associate these features with sound labels. Automatic sound classification has broad applications, including environmental monitoring, medical diagnosis (e.g., respiratory sound analysis), music information retrieval, security systems, and biodiversity research through automated species identification. 1.2.2 Semantics and Acoustic Features In sound classification, both acoustic features of a sound and it’s semantic information such as title, tags and description play a vital role in understanding and categorizing audio content. Acoustic features refer to measurable information extracted from an audio waveform, such as spectral shape, spectral engery, pitch, dynamics or 6Chapter 1. Introduction temporal features. These features are often represented using Mel-frequency cepstral Coefficients (MFCCs), log-Mel spectrograms, or learned embeddings that capture the various properties of a sound and are essential for models that use audio as a basis. On the other hand, semantic features derive from human-annotated metadata like titles, tags and descriptions, often offering context and meaning beyond a raw audio signal. For example, for a sound that has rain noise, a tag like "rain", "water", "rainfall" helps provide meaningful context and disambiguate between acoustically similar sounds. Acoustic features typically exist in high-dimensional spaces and capture temporalspectral patterns, while semantic features operate in conceptual spaces where relationships between categories can be learned through word embeddings or ontological structures. At the same time acoustic features are sensitive to recording conditions, background noise, and signal quality, whereas semantic features remain consistent across different audio recordings of the same conceptual category. This makes semantic features valuable for generalization across diverse recording environments and other variable factors. 1.2.3 Multimodal learning Multimodal learning refers to the process of modeling information from multiple data sources or modalities, in this scenario of sound classification, this would involve acoustic and semantic features of a sound. In multi-modal systems, combining these two modalities, acoustic and semantic features, can enhance classification performance, especially in noisy or ambiguous scenarios where one modality alone may be insufficient By combining these modalities, the system would aim to use complementary strengths where audio features could provide detailed signal-level information and text/semantic features could provide high level information about the audio [6]. Multimodal systems show better robustness compared to unimodal approaches as 1.2. Key Concepts 7 they can maintain performance even when one modality is degraded. For instance if audio quality is poor due to background noise, semantic features from metadata can compensate, and vice versa when textual descriptions are incorrect or ambiguous. 1.2.4 Freesound Freesound is an open, collaborative database platform where users can upload and share audio recordings ranging from everyday sounds to experimental synthesized textures with over 670,000 plus user-contributed audio samples. It serves as a vast library of audio resource for sound designers, researchers and artists. The platform uses advanced technologies for classification of sound, retrieval and search, leveraging the Essentia audio analysis library to automatically extract acoustic features from uploaded sounds and the similarity search engine for content-based retrieval. Additionally, Freesound has recently implemented the Broad Sound Taxonomy (BST)[7] as its organizational scheme. This enables users to find sounds through traditional text-based searches, as well as Query-by-Example functionality, where users can upload audio files to find acoustically similar sounds based on spectral, timbral and other acoustic characteristics. 1.2.5 Universal Category System (UCS) The Universal Category System (UCS) is a standardized heirarchical taxonomy developed to organize and label SFX in professional audio environments. It consists of around 82 top-level categories with over 700 sub-categories, allowing for consistent naming and structure across various levels which enables a uniform structure in commercial sound libraries. 1.2.6 Evaluation metrics and Performance Assessment Performance evaluation in automatic sound classification relies on several key metrics that provide different perspectives on system performance. These metrics are important for assessing the classification performance and accuracy and assess if there are any imbalances. 8Chapter 1. Introduction Accuracy Accuracy represents the proportion of correctly classified samples over the total number of samples: Accuracy =T P +TN TP +TN +FP +F N =Number of Correct Predictions Total Number of Predictions where TP (True Positives), TN (True Negatives), F P (False Positives), and F N (False Negatives) represent the counts from the confusion matrix. Balanced Accuracy Balanced accuracy addresses the limitations of standard accuracy by computing the average recall obtained on each class, providing equal weight to each class (balanced) regardless of its frequency: Balanced Accuracy =1 2TP TP +F N +TN TN +F P  For multi-class problems with Cclasses, this extends to: Balanced Accuracy =1 C C X i=1 TPi TPi+F Ni This metric is particularly valuable for sound classification tasks where certain categories may be underrepresented in the dataset. F1-Score The F1-score provides the harmonic mean of precision and recall, offering a single metric that balances both measures: Precision =TP TP +F P 1.2. Key Concepts 9 Recall =TP TP +F N F1-Score = 2 ·Precision ×Recall Precision +Recall =2·TP 2·TP +F P +FN The F1-score is particularly useful when both false positives and false negatives carry significant costs, making it well-suited for sound classification applications where both precision and recall are important. Weighted F1-Score For multi-class classification problems, the weighted F1-score computes the F1-score for each class and then calculates a weighted average based on the support (number of true instances) for each class: F1-Scoreweighted =1 N C X i=1 ni·F1-Scorei where Nis the total number of samples, Cis the number of classes, niis the number of samples in class i, and F1-Scoreiis the F1-score for class i. This metric accounts for class imbalance by giving more weight to classes with more samples which is important in datasets with class imbalances. Confusion Matrix The confusion matrix provides a comprehensive view of classification performance by showing the actual versus predicted classifications for each class. For a multi-class problem with Cclasses, the confusion matrix Mis a C×Cmatrix where: Mi,j =number of samples with true label ipredicted as label j The confusion matrix enables detailed analysis of classification errors, revealing 10 Chapter 1. Introduction which classes are most commonly confused with each other. For sound classification, this is useful for identifying acoustically or semantically similar categories that the system struggles to differentiate. 1.3 Objectives The main objective of this thesis is to investigate the practical applicability of standardized taxonomies, specifically the Universal Category System (UCS), for ASC in real-world, user generated audio environments. While prior research has demonstrated high classification accuracy in controlled and professionally curated datasets, there is limited understanding of how these systems would perform in communitydriven platforms such as Freesound. This thesis aims to bridge that gap through the following specific objectives: •Enhance and refine already existing classifiers: To improve and refine already existing classifiers like MLP for sound classification systems using additional embeddings and data accordingly for better performance. •Assess domain transfer effectiveness: Evaluate how well UCS-based classifiers trained on professionally curated datasets such as Pro Sound Effects (PSE150K), generalize to user-generated content from Freesound. Analyze the performance degradation caused by domain shift and analyze its implications for classification. •Evaluate the role of multimodal learning: Implement and evaluate unimodal and multi-modal classification architectures to assess how different modalities contribute to predictive performance. Determine the relative contribution of audio and textual metadata in classification tasks and evaluate their effectiveness. •Provide valuable insights for practical deployment: Research and investigate possible directions towards designing more robust and generalizable ASC systems that can work with varied datasets and audio content. 1.4. Structure of the Report 11 •Develop a dataset for cross-domain evaluation: Construct and make publicly available a custom built UCS-organized Freesound dataset that allows for evaluation of domain transfer from professional to user-generated audio content. This dataset contribution aims to enable future research, to conduct standardized comparisons and understanding of domain adaptation challenges in sound classification systems. The custom Freesound dataset and code publicly available1for further research. 1.4 Structure of the Report The remainder of this thesis is organized as follows: Chapter 2: State of the Art - This chapter discusses the history of automatic sound classification systems, from traditional signal processing approaches to deep learning techniques, including current approaches that are used. It also introduces key technologies, taxonomies and models relevant to this study. Chapter 3: Methodology - This chapter outlines the breakdown of the experiments and procedures used throughout this research. Provides explanations of the dataset collection, pre-processing steps, model configurations and evaluation methodologies. Chapter 4: Results - This chapter presents the outcomes of the experiments, including performance metrics and observations. The key findings are highlighted and compared across different models and data conditions. Chapter 5: Discussion - This chapter describes and analyzes the results obtained from all the experiments, while trying to interpret and understand the implications of the results in the context of real-world sound classification. Chapter 6: Conclusion - This final chapter summarizes the main contributions and findings of the thesis. Discussing on the broader significance of the work, along with suggestions for possible future improvements in sound classification systems. 1https://github.com/MadhavJ06/freesound_ucs.git Chapter 2 State of the Art The field of automatic sound classification has evolved significantly over the past two decades, driven by advances in signal processing, machine learning, and the growing availability of large-scale audio datasets. This chapter provides a comprehensive review of the current state-of-the-art in automatic sound classification, with particular emphasis on sound effects classification, taxonomic systems, and multimodal approaches that combine acoustic and semantic features. Finally, we identify possible gaps in the current literature, particularly the lack of systematic evaluation frameworks for cross-domain sound effect classification and the limited exploration of UCS-based taxonomic classification in user-generated content scenarios. This analysis establishes the foundation for the methodological contributions presented in the following chapters and puts this research within the broader context of automatic sound classification and audio content organization. 2.1 History of Sound Classification Early research in sound classification drew from various branches of study such as cognitive sciences and psychoacoustics, where the goal was to understand how humans perceived and distinguished sounds. These early studies focused on Identifying perceptual features such as pitch, timbre and temporal structures which enabled 12 2.2. Current Approaches 13 humans to differentiate sounds. This work laid the foundation for computational models by establishing key acoustic properties that influence auditory perception and classification. Initial works began with a focus on pitch perception for sound classification researches. Helmholtz’s place theory and Terhardt’s harmonic template model[8][9] demonstrated how our brain and auditory system are stimulated differently according to the frequencies and tonal qualities in a sound. Risset and Chowning developed computational models[10][11] that analyzed sounds based on their spectral envelopes. These spectral profiles were used to distinguish between different instruments. Theories of categorization in cognitive psychology have significantly influenced sound classification research. Gestalt’s principles by Wertheimer[12] explain how humans organize auditory stimuli into meaningful patterns, such as grouping environmental sounds into a single category. Rosch’s prototype theory[13] suggests categorization is based on resemblance to a familiar or proper association to a sound, similar to how musical genres are identified and categorized by humans. Bergman introduced Auditory Scene Analysis[14] which describes how humans perceive complex soundscapes by being able to segregate and analyze different auditory elements. These studies provided the conceptual framework and foundations for the development of computational methods to analyze sounds based on their spectral characteristics. 2.2 Current Approaches 2.2.1 Machine learning based approaches In early stages while adopting machine learning (ML) methodologies for sound classification, handcrafted features such as Mel-Frequency Cepstral Coefficients (MFCCs) [15] were widely used for feature extraction. These features formed the basis for traditional machine learning models, including Hidden Markov Models (HMMs), Support Vector Machines (SVMs) and Gaussian Mixture Models (GMMs) 20 Chapter 2. State of the Art be shared across multiple parent categories, creating a flexible vocabulary. For instance, “bells” could be part of “metal” and “music” as well or the subcategory “movement” might describe both “cloth rustling” and “plastic crumpling”, “crackle” could equally apply to the sound of burning fire or sparking electricity. This shared modifier approach efficiently mirrors how sound designers naturally describe effects in real-world practice while preventing redundancy and maintaining consistency. Another lower level of categories present in UCS, paving to the system being more granular in nature is through Category IDs, which represent unique combinations of top-level categories and their associated subcategories. There are 457 distinct identifiers formed by algorithmically merging elements from both tiers of the hierarchy. For example, combining top-level category “Water” with subcategory “flow” will yield the category ID “WATERflow” or “WATflw”. Figure 2: UCS Categorization example This hierarchical structure offers several key advantages for both human users and automatic sound classification systems. As the UCS represents a widely adopted standardization effort, evaluating a classifier performance within it’s framework provides directly applicable insights for various workflows. 2.5 UCS deployment The UCS has been widely adopted globally by professional sound libraries and audio management tools. ProSoundEffects, Soundly Pro, Krotos are libraries that offer extensive industry standard libraries (e.g. BBC sound effects, Hollywood edge) that 2.5. UCS deployment 21 contain a diverse range of SFX for commercial use. These libraries are aligned with metadata according to the UCS taxonomy for precise search-ability. As highlighted by Alison et al.[42] even UCS-compliant systems face challenges. The study addresses the challenges of inconsistent metadata and taxonomy updates in SFX libraries by proposing context-agnostic audio embeddings trained via representation learning. These embeddings capture acoustic properties and features independent of predefined labels. This was seen to outperform traditional methods like OpenL3 by handling class imbalance and label noise through cross-dataset training and metric learning. While the UCS standardizes libraries, this work complements the system by creating generalized embeddings applicable to both UCScompliant and non-compliant datasets. Although, this approach does not assign fixed labels like traditional classifiers, instead it retrieves based on acoustic similarity. This approach in a multimodal methodology could be relevant for platforms like Freesound, where user-generated tags often deviate from standards, and provides a proper framework for evaluation. By adopting the hierarchical taxonomy of UCS as a classification benchmark, our research can address the challenges of classification and taxonomies in Freesound, where inconsistent metadata currently limit search-ability and usability. The deployment of UCS in a real world context, such as the Freesound community-driven tagging system, will help to address many challenges. Chiruthapudi, S. [43] explored sound classification using the Universal Category System (UCS) on the ProSoundEffects dataset, developing and evaluating automatic classifiers based on standard classification metrics to assess their performance. This research extends the previous work by trying to refine and improve the classifiers and evaluating how established UCS classifiers perform under imperfect real-world conditions. By implementing UCS-based classification with Freesound data, we aim to evaluate and research on how community based platforms could potentially benefit from Industry standards, while simultaneously providing further insights and research opportunities into how these taxonomies perform. Chapter 3 Methodology This chapter outlines the methodology used in this thesis to experiment and investigate sound classification using UCS[2] in the context of Freesound. The pipeline consists of comparing different models, development of Freesound dataset, feature extraction from the datasets, evaluation and possible refinements and enhancements of existing classifiers using different architectures, and cross-domain evaluation. The experiments are performed on various levels starting from a small subset of the PSE150K, using which we evaluate various architectures and try to find the most optimal architectures for the experiments. A set of different models were obtained from these evaluations with different multimodal features which were used to finally evaluate cross-domain performance. 3.1 Dataset 3.1.1 PSE150K The PSE150K dataset is an audio sound effects collection that was developed by Pro Sound Effects (PSE) company. PSE has developed one of the largest library of professionally recorded sounds. For our work we use a dataset that was provided by PSE to the Music Technology Group (MTG) at Pompeu Fabra University[44]. This dataset in total consists of 380,000 audio files which have been categorized according 22 3.1. Dataset 23 to the Universal Category System (UCS)[2] of 82 top level categories and 753 sub categories. From this full dataset, we use approximately 143,300 audio files along with which a metadata file of .csv format that contains the mapping of each audio file to its corresponding top-level category, sub-category and category ID which we call as the PSE150k dataset. From dataset analysis, it was noted that across the top level 82 categories, there were 368 unique subcategories. The dataset shows significant variability in category representation, ranging from 43 samples (CERAMICS) to 10,539 samples (VEHICLES). The most populated categories are common sound effects like VEHICLES, AMBIENCE, VOICES, WATER, ANIMALS, while some ambiguous and specialized categories contain fewer samples reflecting the more niche applications in professional usage. Since 143,000 audio files will be computaTable 1: Summary of Dataset Statistics for PSE150K and PSE8K Dataset Top-level Categories Sub-categories Avg. Samples. PSE150K 82 368 ≈1747 PSE8K 82 291 ≈97 tionally expensive to perform evaluations on, we use a smaller subset of this dataset of approximately 8,000 audio files were used to perform a preliminary rapid evaluation of sound classification systems. This subset PSE8K is balanced targeting approximately 100 samples per category across all the 82-top level UCS categories. 74 categories out of the 82 has 100 samples each with some categories varying with lesser numbers like WINDOWS (62 samples), EQUIPMENT(59 samples) etc. This dataset has better class distribution and was used as the base for robust classification model training and evaluations. This dataset serves as the primary dataset representing a professionally curated and controlled dataset. 3.1.2 Freesound Data Initial Dataset development and limitations A custom Freesound dataset was created specifically for cross-domain evaluation in this research. The initial phase involved developing a smaller dataset comprising 24 Chapter 3. Methodology approximately 1,534 audio clips sourced through the Freesound API. Each audio sample was mapped to the 82 top-level UCS categories and accompanied by comprehensive metadata including Freesound ID, titles, tags, descriptions, and source URLs. The dataset targeted approximately 20 samples per category, using generic search queries that combined top-level category names with relevant synonyms to ensure broad coverage across each class. However, several limitations emerged during the data collection process. Since the dataset exclusively used only the MP3 files from Freesound rather than files of all formats, the collected data was constrained in both quality and availability. This limitation, combined with the generic nature of search queries, resulted in insufficient samples for some ambiguous top-level categories, ultimately reducing the final dataset to 79 categories out of the original 82. Further analysis revealed significant inconsistencies in audio quality across samples, showing the diversity of recordings and environments used by different Freesound contributors. Additionally, the absence of user-based sampling functionalities led to an unintended bias where multiple samples were collected from the same users, potentially compromising the dataset’s diversity and representativeness for robust cross-domain evaluation. Comprehensive Dataset creation In order to mitigate these issues with the preliminary dataset, we set out to build a larger dataset that could be used as a base for future work in cross-domain classification from professional datasets to more real-world user-generated content datasets. A comprehensive Freesound dataset was developed for cross-domain evaluation, comprising approximately 9,400 audio files mapped to the 82 top-level UCS categories. The dataset employed a query-based collection methodology utilizing the Freesound API, targeting around 100-120 samples per category. For each UCS category, multiple queries were created using the top-level category as the primary query and subcategories and their synonyms as secondary queries. Each query was structured in the following manner: "<subcategory/synonym> <primary query>". This made the queries very extensive in nature, going through various different possibilities in 3.1. Dataset 25 each top-level category. The number of samples to be collected from a category was distributed across the queries depending on the number of queries that were present for that respective category. For example, if a category "AIR" had 10 queries, each query would collect 12 samples. These queries were created and mapped across all the categories manually using the UCS[2] categorization available on their website. In order to ensure proper distribution among a category with different queries, fallback strategies were implemented such that if one or more queries did not collect a sufficient amount of samples, the collection was redirected towards the other search queries that were more successful. This process was repeated until each category collected a satisfactory amount of samples. Additionally, in order to ensure quality and prevent bias in the dataset, some filtering functionalities were implemented. The API allows for only downloads of the previews of the audio files of full length and not the original files in the original format, so preference for high-quality formats was given (WAV > FLAC > MP3), and the preview files were downloaded in .ogg format, which is of higher quality than MP3, the files chosen were filtered to have minimum user ratings of 3 or greater wherever available. In order to prevent bias, global tracking of samples per user across the collection was implemented so that no more than 3 or 4 samples were collected per user across the different categories. This ensured acoustic and metadata diversity in the dataset. Along with the collection of the files, the metadata for each audio file was also collected and stored, which includes the file name, Freesound ID, Freesound URL, UCS category of the respective file, query used, tags, description, uploader information, rating, and audio file information (file format, sample rate). Duplicates were also not allowed to be downloaded. This whole process was automated in a script with oversight and manual supervision of the data collected. The resulting dataset achieved approximately 120 samples per category, with some ambiguous or specialized categories yielding fewer samples due to limited availability of suitable content on the platform as shown in 3. This comprehensive dataset represents a significant improvement over the preliminary version, establishing a robust and strong foundation for evaluating cross-domain automatic sound classification 26 Chapter 3. Methodology Figure 3: Category Distribution in Freesound dataset performance between professionally curated sound libraries and user-generated audio content from collaborative platforms. A notable variation emerged between the professional PSE dataset and the usergenerated Freesound dataset across data quality and consistency. The audio quality in the Freesound dataset exhibited significant variability across different contributors, reflecting the diverse recording equipment, acoustic environments, and also expertise of individual users. This contrasts with the PSE dataset’s more consistent nature in it’s samples. The PSE dataset generally contains more consistent types of recordings and quality of audio across the board. A key distinction compared to the other dataset may lie more in the use of manual labeling rather than automatic querying. Similarly, the metadata quality revealed significant differences between the two datasets. While the PSE dataset maintains comprehensive and standardized metadata with systematic file naming conventions and structured tagging methodologies, 3.2. Feature Extraction 27 the Freesound dataset’s user-contributed nature showed highly inconsistent metadata. This inconsistency was reflected throughout the dataset, some users would tag their files with extensive, detailed descriptions while others provided a few words as descriptions. File names ranged from alphanumeric codes to overly verbose descriptions. This stark contrast between the professional, standardized PSE dataset and the chaotic reality of user-generated content perfectly illustrates one of the biggest challenges in audio classification research: training models on professional, curated data only to deploy them in the unpredictable world of user-contributed uploads. 3.1.3 Other datasets In order to compare UCS with other taxonomies, two other datasets were evaluated - BSD10K (Broad sound Dataset)[4], which contains 10,309 Freesound audio clips, manually annotated according to the Broad Sound Taxonomy which has 5 top-level categories and 23 subcategories. Another dataset that was evaluated is the FSD50K (Freesound Dataset 50K)[39], which contains 51,197 Freesound audio clips that are assigned to different classes aligned with the AudioSet ontology[5]. The AudioSet ontology is a hierarchical taxonomy of 632 sound event classes, developed by Google. 3.2 Feature Extraction Feature extraction for the datasets were done using LAION CLAP[34]. The CLAP embeddings were used to represent both audio and textual data in a common latent space. The CLAP model provides a 512-dimensional embedding for each audio clip and corresponding text metadata. CLAP embeddings were extracted directly from the waveforms of the audio files using a pre-trained LAION-CLAP encoder for PSE150K and the Freesound data alike. The processing chain begins with audio loading via librosa[45] at a target sampling rate of 48kHz, followed by conversion of the audio into mel spectrograms which is handled by the CLAP model. Each audio file is passed through CLAP’s audio encoder to produce a 512-dimensional feature vector that captures semantic audio representations. In order to experiment and investigate different modalities, text metadata was pro- 28 Chapter 3. Methodology cessed and extracted from the CSV file that had information about each audio file along with it’s file name, top-level categories and subcategories. For the PSE150K dataset, the metadata was professionally curated and had proper structure, whereas for real-world data like Freesound, the metadata ranged from detailed descriptions and tags to minimal words or incomplete descriptions. The text feature extraction generated descriptions for audio samples. The meaningful keywords from filenames and descriptions were extracted and created into structured descriptions such as "Audio sample with characteristics (keywords) and type (subcategory)" along with some simpler tags, the top level categories and subcategories. These generated text descriptions are passed through CLAP’s text encoder to produce 512-dimensional embeddings that share the same space as the audio embeddings. This enabled multimodal comparisons. The system employs two multimodal fusion strategies to combine audio and text embeddings. Concatenation fusion merges the 512-dimensional audio and text vectors into a 1024-dimensional representation, preserving all information from both modalities. Weighted fusion maintains the original 512-dimensional space by combining modalities with learnable weights, defaulting to 80% audio and 20% text (alpha = 0.8). The same feature extraction pipeline was used on both the PSE150K and Freesound dataset The subset PSE8K consisted of 7990 audio files in total, and to ensure consistency and completeness, the file names without their extensions were cross-referenced and checked with the files available according to the metadata file from the embeddings extracted from CLAP. This matching procedure revealed that valid embeddings were available for 5,568 audio files within the subset. Since this subset was derived from the bigger full data set of 350K audio files, there ended up being some files that did not have the extracted embeddings. However it was confirmed that more or less the categories were still well balanced and not substantially skewed. 3.3. Classification with dataset subset (PSE8K) 29 3.3 Classification with dataset subset (PSE8K) 3.3.1 Experimental Design and Model Implementation To establish baseline performance and explore optimal architectures, initial experiments were conducted on the PSE8K subset, which provided a suitable foundation for rapid evaluation and iterative model development. Several classification models were tested to assess their ability to handle the unique challenges posed by both audio and text data, as well as their combined multimodal embeddings. The first model evaluated was the K-Nearest Neighbors (KNN) classifier, implemented, serving as a non-parametric baseline model for classification evaluations, where classification decisions are made by finding the k= 3 nearest neighbors of a query point in the embedding space based on cosine distance. Majority voting among these neighbors determines the predicted class. To handle the high-dimensional CLAP embeddings effectively, StandardScaler normalization was applied to standardize the feature space before computation. This model supports multimodal fusion by concatenating audio and text embeddings into a 1024-dimensional vector but also uses audio-only or text-only modes using 512-dimensional inputs. The KNN approach does not involve an explicit training phase but provides valuable insight into the separability of the embedding space and robustness against class imbalance through local majority voting. The next model implemented was the Multi-Layer Perceptron (MLP), implemented with feedforward neural network, featuring two hidden layers containing 1024 and 512 neurons, respectively, and applies the tanh activation function for non-linearity. The model training leverages the Adam optimizer with an adaptive learning rate scheduler initialized at α= 0.001. L2 regularization with penalty weight λ= 0.001 reduces overfitting, the model training used early stopping based on validation accuracy for better convergence control. Input features are preprocessed through StandardScaler, while categorical targets utilize Label Encoder encoding. The MLP also similarly supports the audio-only, text-only and the multimodal con- 36 Chapter 4. Results mance. The evaluation used standardized feature extraction pipelines, employing the CLAP[34] model to generate audio and text embeddings, which served as inputs for various classification architectures. This approach enabled rapid benchmarking of different modality-specific and multimodal fusion models under a consistent framework, yielding initial performance results which help guide the subsequent tasks in the project. 4.1.1 Audio-Only Configuration Table 2: Audio-Only Model Performance on PSE8K Model Type Accuracy Balanced Acc F1 Macro F1 Weighted KNN 0.5202 0.4885 0.4554 0.4913 MLP 0.5701 0.5142 0.4942 0.5499 Transformer 0.5393 0.4829 0.4682 0.5245 Standard CA 0.5509 0.5042 0.4845 0.5365 Patch CA 0.5298 0.4818 0.4631 0.5134 Best Audio-Only: MLP (Accuracy = 0.5701) 4.1.2 Text-Only Configuration Table 3: Text-Only Model Performance on PSE8K Model Type Accuracy Balanced Acc F1 Macro F1 Weighted KNN 0.8431 0.8432 0.8427 0.8410 MLP 0.9097 0.9106 0.9106 0.9092 Transformer 0.9069 0.9079 0.9078 0.9074 Standard CA 0.9042 0.9045 0.9034 0.9036 Patch CA 0.8972 0.8984 0.8976 0.8966 Best Text-Only: MLP (Accuracy = 0.9097) 4.1. Rapid Evaluation Results 37 4.1.3 Multimodal Concatenation Table 4: Multimodal Concatenation Model Performance on PSE8K Model Type Accuracy Balanced Acc F1 Macro F1 Weighted KNN 0.7965 0.7572 0.7466 0.7854 MLP 0.9002 0.8773 0.8733 0.8941 Transformer 0.9079 0.8844 0.8760 0.9057 Standard CA 0.9021 0.8797 0.8784 0.8963 Patch CA 0.8983 0.8663 0.8647 0.8944 Best Multimodal Concatenation: Transformer (Accuracy = 0.9079) 4.1.4 Multimodal Weighted Fusion (alpha=0.8) Table 5: Multimodal Weighted Fusion Model Performance on PSE8K Model Type Accuracy Balanced Acc F1 Macro F1 Weighted KNN 0.5739 0.5425 0.5090 0.5486 MLP 0.6833 0.6379 0.6346 0.6699 Transformer 0.6583 0.6064 0.5807 0.6409 Standard CA 0.8848 0.8617 0.8600 0.8776 Patch CA 0.8983 0.8775 0.8760 0.8925 Best Multimodal Weighted Fusion: Patch Cross-Attention (Accuracy = 0.8983) 4.1.5 Model Performance Analysis Audio-Only Embeddings Among the audio-only models, the MLP) achieved the highest accuracy at 57.01%, with a balanced accuracy of 51.42%, outperforming other architectures including K-Nearest Neighbors (KNN) and Transformer variants. The MLP also recorded the leading F1 scores, with a Macro F1 of 49.42% and a Weighted F1 of 54.99%, indicating its better capability in capturing prominent audio 38 Chapter 4. Results features from the CLAP embeddings. While the KNN model showed the lowest performance with an accuracy of 52.02%. Text-Only Embeddings Text-based models demonstrated significantly higher performance than audio-only counterparts. The MLP model again achieved the highest with an accuracy of 90.97%, balanced accuracy of 91.06%, and high Macro and Weighted F1 scores nearing 91%, showcasing the model’s ability to learn from metadata and the textual description features. Transformer and Cross-Attention models also performed similarly well with negligible difference compared to the MLP, achieving accuracies in the 90% range. The KNN model, while lowest among these, still reached an accuracy of 84.31%. Figure 4: Balanced Accuracy across different models Multimodal Concatenation Combining audio and text embeddings via simple concatenation improved overall performance across all models. The Transformer model topped this group with a 90.79% accuracy and balanced accuracy of 88.44%, followed closely by MLP and Cross-Attention models. The KNN model also showed gains over audio-only mode, reaching nearly 80% accuracy, possibly indicating some advantage in using multimodal embeddings. Multimodal Weighted Fusion (alpha=0.8) When applying weighted fusion with a dominant audio weight of 0.8, the Cross-Attention models performed best by 4.2. Evaluation of full dataset 39 a notable margin. Patch Cross-Attention attained an accuracy of 89.83%, balanced accuracy of 87.75%, and F1 scores above 87%. The Standard Cross-Attention model similarly recorded high scores around 88% accuracy. Other models including MLP and Transformer saw decreased accuracies with 68% and 65% respectively compared to concatenation. KNN demonstrated its lowest performance here, around 57% accuracy. Overall, these results demonstrate that the MLP consistently achieved the best performance despite its relatively simpler architecture compared to more complex models such as Transformers and Cross-Attention networks. This trend held true across most feature configurations, with the exception of the weighted multimodal fusion setting, where the cross-attention models exhibited stronger performance. Further details and analysis of this exception are provided in the Discussion section. 4.2 Evaluation of full dataset The full PSE150K dataset was evaluated using the best-performing MLP model identified from the subset experiments across different embedding configurations. The training and evaluation in this section are conducted exclusively with MLP architectures, leveraging their demonstrated effectiveness. This approach allows us to use on the strengths of the MLP model while scaling evaluation to the entire dataset, providing a comprehensive assessment of model performance on a broader dataset. The trained MLP models will be further utilized for cross-domain evaluation tasks, including experiments on the Freesound dataset. Table 6: PSE150K Model Performance (MLP Architectures) Model Accuracy Bal Acc F1 Macro F1 Weighted Text-Only 0.9833 0.9796 0.9805 0.9833 Multimodal Concat 0.9838 0.9808 0.9810 0.9837 Multimodal Weighted 0.8344 0.8070 0.8037 0.8335 Audio-Only 0.6746 0.6270 0.6409 0.6720 The performance of the MLP model evaluated across four embedding configurations 40 Chapter 4. Results on the full PSE150K dataset is presented in 6 and visualized in the confusion matrices shown in Figure 5. The multimodal concatenation configuration yielded the highest accuracy at 98.38%, with corresponding balanced accuracy and F1 scores exceeding 98%. The text-only configuration showed high accuracy as well, with an accuracy of 98.33% and balanced accuracy of 97.96%. In both these settings, the confusion matrices (Figure 5b and 5c) display strong diagonal dominance, indicating near perfect classification across nearly all categories; however, this displayed performance may overestimate the model’s true effectiveness given the complexity of the classification task and might not fully reflect the challenges of the task, and the results should be interpreted with caution. In contrast, the multimodal weighted fusion configuration, employing an audio weight of 0.8, resulted in reduced performance, with accuracy and balanced accuracy values around 83%. The corresponding confusion matrix (Figure 5d) reveals an increase in off-diagonal errors, suggesting higher misclassification rates compared to concatenated or text-only embeddings. The audio-only configuration produced the lowest metrics, recording an accuracy of 67.46% and a balanced accuracy of 62.70%. Its confusion matrix (Figure 5) further illustrates a significant presence of off-diagonal elements, highlighting increased confusion between categories when relying solely on audio features, which more accurately reflects the challenges typical of audio classification tasks. Collectively, these results provide valuable insight into the behavior of the different embedding modalities. The audio-based CLAP embeddings appear to offer a more realistic representation of the classification challenges inherent in audio data, while the text embeddings tend to produce more optimistic performance estimates that may overstate model capabilities. Building on these observations, the trained models were applied to cross-domain evaluation on the Freesound dataset to further assess their generalization capabilities beyond the original training distribution. 4.2. Evaluation of full dataset 41 (a) (b) (c) (d) Figure 5: Confusion matrices for MLP evaluation of (a) Audio-only, (b) Text-only, (c) Multimodal concatenated, and (d) Multimodal weighted configurations. 42 Chapter 4. Results 4.3 Evaluation of PSE trained models on Freesound Data The PSE-trained models were evaluated on two versions of the Freesound dataset: a preliminary subset and an extended, more comprehensive collection. This evaluation involved a number of different experiments designed to compare model performance and provide detailed analysis across different embedding configurations. The aim was to assess the generalization capability of the trained models when applied to external, real-world audio datasets with varying characteristics. 4.3.1 Results on Preliminary Freesound Dataset The evaluation results on the preliminary Freesound-UCS dataset are described below. The overall model performance across all embedding modalities and fusion strategies is significantly lower compared to the PSE150K evaluations, reflecting the increased complexity and variability of the Freesound data. Among the different configurations, the multimodal concat. model achieved the highest accuracy of 16.71%, alongside balanced accuracy and F1 metrics exceeding 15%. The audio-only model also performed competitively, with an accuracy of 16.02%, marginally below the multimodal concatenation. The multimodal weighted fusion and text-only models yielded slightly lower accuracies, around 15.5% and 12.0% as the lowest for the text-only model respectively. This preliminary evaluation provides a valuable baseline for further development of an extended freesound dataset with a more diverse sample set, allowing for more comprehensive and reliable evaluation of model performance. 4.3.2 Results on Extended Freesound Dataset The extended Freesound UCS dataset used for evaluation comprises approximately 9,400 audio files across the 82 top-level UCS[2] sound categories, with an average of about 120 samples per category. This dataset offers a diverse and representative 4.3. Evaluation of PSE trained models on Freesound Data 43 collection of real-world audio samples, making it a challenging benchmark for largescale, multi-class sound classification. The evaluation results as shown in 7 reveal that model performance on this dataset, while modest, reflects the inherent complexity and variability within Freesound samples. Among the four embedding configurations tested, the multimodal weighted fusion approach achieved the highest accuracy of approximately 28.7%, balanced accuracy around 28%, and F1 weighted score close to 26.8%. Multimodal concatenation followed with comparable but slightly lower performance, with accuracy around 26.1%. The audio-only model showed moderate effectiveness with an accuracy near 23.8%, and the text-only model recorded the lowest metrics, with accuracy slightly above 20.1%. Table 7: Freesound-UCS Evaluation Results Summary Model Accuracy Bal Acc F1 Macro F1 Weighted Audio-Only 0.2383 0.2323 0.2209 0.2262 Text-Only 0.2057 0.1974 0.1902 0.1971 Multimodal Concat 0.2602 0.2507 0.2412 0.2484 Multimodal Weighted 0.2872 0.2801 0.2688 0.2751 These results seem to indicate a trend where combining audio and textual features seems to enhance classification capability despite the dataset’s complexity.As shown by the confusion matrices in Figure 6, all model configurations exhibit substantial off-diagonal errors, highlighting frequent misclassifications across many of the 82 toplevel categories. While the multimodal approaches outperform single-modality models, the absolute scores remain modest, indicating persistent challenges in achieving robust generalization. The relatively lower scores across all models point out the major challenge of generalization when moving from controlled and professional datasets to a diverse and varied heterogeneous collection like Freesound UCS Dataset. The rich diversity of sounds, annotation variability, and domain mismatch between training and testing data likely contribute to these outcomes. Overall, these findings provide a comprehensive perspective on model performance 44 Chapter 4. Results (a) (b) (c) (d) Figure 6: Confusion matrices for evaluation on Freesound dataset (a) Audio-only, (b) Text-only, (c) Multimodal concatenated, and (d) Multimodal weighted configurations. 4.4. Fine-tuning Results 45 and generalization under realistic automatic Sound classification conditions. The results and visual analysis suggest key directions for future research, particularly in refining embeddings, improving fusion strategies, and addressing domain adaptation challenges for large-scale, multi-class sound recognition. Taken together, the results provide a foundation for a lot of interpretation of the model behavior in sound classification and the broader implications of cross-domain sound classifications. These aspects will be explored further in greater detail in the following Discussion section. 4.4 Fine-tuning Results Fine-tuning consistently improved model performance across all configurations and training data percentages, the accuracies of different training data used for the fine-tuning is shown in figure 7. With just 10% of Freesound UCS data, each model slightly outperformed its original PSE baseline, and more improvements were observed as more training data was used. The audio-only model accuracy increased moderately, from 24.3% at 10% data to 25.0% at 50%. For the text-only model, gains were a bit more pronounced, jumping from 21.2% to 23.7%. Multimodal models achieved the best results overall: the concatenation model rose from 26.8% to 29.0%, and the weighted multimodal model achieved the highest accuracy, from 29.6% to 30.2% as training data increased from 10% to 50%. These findings demonstrate that fine-tuning is especially effective for multimodal systems and that performance benefits scale with the size of the target dataset. 52 Chapter 5. Discussion embeddings from CLAP [34]. CLAP embeddings themselves are inherently multimodal in nature as CLAP is trained to capture both acoustic and semantic information. However the CLAP [34] embeddings gave only an accuracy of 23% which is lesser than the mutlimodal fusion models. This performance boost likely stems from the additional cues provided by the training on the text metadata, which can supplement audio-derived representations in the cases where textual or acoustic features alone are insufficient. However, overall the results are still nowhere near ideal for the task of Automatic Sound Classification and underscores the challenges in generalization of sounds when the data is unstructured, varying in quality and noisy in nature. The confusion matrices (see figure 6 highlight how there is still substantial misclassifications, showing how there’s high category overlap and the nuances of real-world sounds. 5.4.1 t-SNE Analysis of Embedding Spaces Across Domains Figure 8 presents a t-SNE visualization comparing PSE (blue) and Freesound (red) embeddings across audio-only, text-only, multimodal concatenation, and weighted multimodal configurations. This analysis offers an intuitive window into the distributional properties and domain overlap of each embedding type. Audio-only (Top Left) t-SNE plot shows pretty high overlap and closer clusters between the PSE [44] and Freesound [1] embeddings, with some clear regions existing where dataset-specific clusters remain. This suggests CLAP audio embeddings capture some similar features, and acoustic variability between curated (PSE) and user-generated (Freesound) data produces not much separation. The somewhat intermixed pattern indicates that audio embeddings provide limited domain generalization, echoing observed model performance. However, the overlap does not necessarily indicate that the embeddings align meaningfully with the intended semantic categories. Rather it may simply reflect low-level acoustic similarities while differences in other factors may remain underrepresented 5.4. Comprehensive Dataset Evaluation 53 Figure 8: t-SNE analysis of PSE vs Freesound Embeddings Text-only (Top Right). For text-only embeddings, the separation between the two datasets is much more pronounced. PSE samples cluster tightly towards the center, reflecting clean and uniform metadata, while Freesound points are more distributed, indicative of noise, heterogeneity, and inconsistent metadata. This explains the significant domain gap and the lower performance with the text-only model. Multimodal Concatenation (Bottom Left). The multimodal concatenated space exhibits more overlap than text-only, yet some distinction between domains persists. Here, the combined cues from audio and text bring the two domains closer, but simple fusion does not completely align the respective embedding distributions. Weighted Multimodal Fusion (Bottom Right). The t-SNE plot for weighted multimodal embeddings (with audio weight α= 0.8) resembles the audio-only configuration, showing increased mixing of the two datasets. While more domain overlap is achieved, clear separation remains, signifying that even strong audio fusion cannot fully bridge domain gaps present in real-world data. Overall, the t-SNE analysis visualizes the challenges of domain adaptation: only partial embedding overlap is achieved across configurations, with audio or multimodal 54 Chapter 5. Discussion representations providing the greatest (but still incomplete) generalization. It is to be noted that the analysis of Freesound vs PSE was done using only 10,000 PSE samples out of the full dataset and the full Freesound dataset, in order to maintain comparability. 5.4.2 Category Analysis: Performance Across Sound Classes To further understand model strengths and weaknesses, a detailed category-wise analysis was conducted, as depicted in Figures 9 and 10. Figure 9 presents the accuracy scores of audio-only, text-only, and multimodal models across all categories. The plot reveals variation in performance between categories and modalities. Some categories such as Fireworks,User Interface, and Water exhibit strong performance, often showing clear benefits from multimodal integration (green bars). In contrast, several categories (e.g., Natural Disaster,Destruction, Archived) show consistently low accuracy across all configurations, suggesting that both modalities struggle to capture discriminative features for these more ambiguous classes. It is to be noted that the "archived" category in the UCS system is a special top-level label meant for sound files that are stored or set aside, not classified by semantic or audio characteristics, making it not so relevant for our current study. The audio-only and multimodal fusion modalities overall show higher percategory performance across the board while the text-only embeddings show good performance only in certain categories like Cartoon and Cloth. Figure 10 highlights this further by relating multimodal accuracy to the number of samples per category and annotating categories based on which modality dominated performance. Categories marked in blue (audio dominant) demonstrate that acoustic features are the main contribution to correct predictions, while red (text dominant) points indicate categories where metadata and textual cues give models an edge. Green points (multimodal benefit) illustrate classes where fusing modalities yields the best results. Many categories, especially those with larger sample sizes, benefit from multimodal approaches, yet a sizable number of classes remain challenging, regardless of modality or dataset size. 5.4. Comprehensive Dataset Evaluation 55 Figure 9: Category-wise accuracy for audio-only, text-only, and multimodal models. Figure 10: Multimodal accuracy vs. sample count by category. 56 Chapter 5. Discussion Taken together, these analyses reveal several key trends: (1) classification success is highly category-dependent, (2) the better performance from multimodal fusion are not uniform and depend on the complementary nature of audio and textual information per class, and (3) for especially difficult or ambiguous categories, neither modality nor increased data alone solves the challenge, pointing to the need for richer representations or targeted data augmentation. This analysis illustrates the potential and limitations of current multimodal systems, possibly guiding future work towards more robust strategies. 5.5 Fine-Tuning Analysis Figure 11: Comparison of Fine-tuned Models Based on Accuracy. Fine-tuning was done on the PSE trained models with 3 different sets of dataset taken from the comprehensive Freesound dataset - 10%, 25%, 50%. Here the plots 11 12 impact of fine-tuning the models. As shown in Figure 11 even modest increments in the amount of data yield measurable improvements over the baseline performance for all the model configurations. Multimodal concatenation and text-only models benefit the most, with gains exceeding 2-3% at the largest finetuning split, while audio-only and multimodal weighted models see more modest but consistent improvement. 5.6. Future Work 57 Figure 12: Fine-tuning Accuracy Improvement Trends. The line plot of accuracy gains points out the importance of using slightly larger fine-tuning sets. All modalities display steadily increasing improvements, with the largest effect seen for configurations that use "more" textual data. This reinforces how much the difference in textual metadata is present between the two datasets PSE[44] and Freesound[1]. This points out the significance of ideally employing both professionally curated datasets and more varied, real-world datasets for domain adaptation, which will enable models to better generalize between them and remain robust in practical applications where data, that is sounds in this case come in all shapes and sizes. 5.6 Future Work This study highlights key directions for advancing automatic sound classification: •Improved Embeddings: Current audio and text embeddings capture limited real-world variability. Future work should focus on developing more robust, semantically rich representations to enhance generalization. Possibly experimenting using other feature extractors other than CLAP.[34] 58 Chapter 5. Discussion •Advanced Fusion Techniques: While multimodal fusion improves accuracy, more dynamic and adaptive fusion methods are needed to better leverage modality-specific strengths on a per-sample basis. •Diverse and Realistic Data: Evaluation on heterogeneous datasets like Freesound UCS is a vital and crucial step in creating robust Automatic Sound Classification Systems while also being able to understand well-structured data as well. Expanding dataset diversity and quality will help models better reflect real-world complexities. •Domain Adaptation and Robustness: Addressing domain mismatch, annotation noise, and acoustic diversity remains a challenge. Incorporating domain adaptation, label alignment, and contextual supervision will be important for scalable sound classification systems. •Category-aware Modeling: Although a complex and difficult task, developing category-specific strategies and targeted data augmentation may further enhance performance on challenging or underrepresented sound classes. Chapter 6 Conclusion This thesis set out to investigate the challenges of automatic sound classification when moving from controlled, professionally curated datasets to the messier realities of real-world data. On the curated professional dataset, audio embeddings and audio-text models performed strongly, showing the promise of current architectures. Yet when applied to the more varied and unpredictable Freesound dataset, performance dropped noticeably, pointing out the obvious issue of domain-gaps. Semantic audio embeddings showed modest but realistic performance, while multimodal fusion (audio+text) configurations performed relatively better. By contrast, text-only embeddings, though overly optimistic on the professional dataset, failed substantially on Freesound data. To explore this further, fine-tuning on different amounts of Freesound data showed clear improvements, especially for multimodal models, highlighting how critical adaptation is when facing real-world variability. Together, these findings suggest that curated datasets provide a strong foundation, but true robustness comes from handling noisy, diverse data to better prepare models for the complexity of everyday audio. Ultimately, this thesis contributes to the ongoing challenges in sound classification and emphasizes the need for robust embeddings and domain-aware training strategies as essential steps toward building scalable, transferable, and reliable Automatic Sound FX Classification Systems for real-world use 59 60 LIST OF FIGURES List of Figures 1 CLAP (Contrastive Language-Audio Pre-training) Architecture . . . 17 2 UCS Categorization example . . . . . . . . . . . . . . . . . . . . . . . 20 3 Category Distribution in Freesound dataset . . . . . . . . . . . . . . . 26 4 Balanced Accuracy across different models . . . . . . . . . . . . . . . 38 5 Confusion matrices for MLP evaluation of (a) Audio-only, (b) Textonly, (c) Multimodal concatenated, and (d) Multimodal weighted configurations. ................................ 41 6 Confusion matrices for evaluation on Freesound dataset (a) Audioonly, (b) Text-only, (c) Multimodal concatenated, and (d) Multimodal weighted configurations. . . . . . . . . . . . . . . . . . . . . . 44 7 Fine-Tuned model accuracies . . . . . . . . . . . . . . . . . . . . . . . 46 8 t-SNE analysis of PSE vs Freesound Embeddings . . . . . . . . . . . 53 9 Category-wise accuracy for audio-only, text-only, and multimodal models. .................................. 55 10 Multimodal accuracy vs. sample count by category. . . . . . . . . . . 55 11 Comparison of Fine-tuned Models Based on Accuracy. . . . . . . . . . 56 12 Fine-tuning Accuracy Improvement Trends. . . . . . . . . . . . . . . 57 List of Tables 1 Summary of Dataset Statistics for PSE150K and PSE8K . . . . . . . 23 2 Audio-Only Model Performance on PSE8K . . . . . . . . . . . . . . . 36 3 Text-Only Model Performance on PSE8K . . . . . . . . . . . . . . . . 36 4 Multimodal Concatenation Model Performance on PSE8K . . . . . . 37 5 Multimodal Weighted Fusion Model Performance on PSE8K . . . . . 37 6 PSE150K Model Performance (MLP Architectures) . . . . . . . . . . 39 7 Freesound-UCS Evaluation Results Summary . . . . . . . . . . . . . 43 61