Pairing Disparate Datasets for Generative Outputs Across Music and Architecture

Pairing Disparate Datasets for Generative Outputs Across Music and Architecture Rolandas Markevi!ius[0009→0003→2402→7320]ω [email protected] Abstract. Most multimodal models utilise semantically consistent data samples for model training and generative outputs. However, for design and composition work, semantic correlations are not the only means for cross-modal translations as suggested by synaesthetic phenomena. In this paper, we propose several methods to pair disparate datasets for training a model to translate between musical compositions and architectural design. We adopt and augment our own datasets to extract high-level features for both music and architectural samples. We utilise self-organising maps and agglomerative clustering strategies to pair data which is then used to train a conditional GAN. The techniques are evaluated through qualitative and quantitative assessment and are also applied in generating architectural compositions. Our experiments suggest promising initial results for feature pairing using hierarchical agglomerative clustering when generating images. We conclude by discussing the meaning and utilization of our proposed techniques as a means to systematically design with abstract features across modalities. Keywords: cross-modal translation ·multimodal design ·data pairing ·generative adversarial networks ·computational design. 1Introduction Research regarding multimodal models has undergone major development over recent years and has dramatically expanded generative capabilities for translation between domains. Models adopting di"usion decoder based techniques have excelled at translations between text and image formats [15,9,8, 4] while more recent LLM based foundation models are now extended to function bi-directionally across domains including text, audio, video and images [11,6]. Most of these developments are focused on performing semantic based translations and require large-scale, well-annotated and aligned datasets. In these cases relationships between modalities are pre-determined by natural multivalence established through ωThe author thanks Roberto Bottazzi and Tom Holberton for their insightful discussions when developing the work as part of the Advanced Architectural Thesis and Advanced Architectural Design modules for the Architecture MArch (Part II) program at The Bartlett School of Architecture (University College London) 2021-2022. The author also thanks the reviewers for their valuable feedback. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 661 R. Markevi!ius association in common context or a single state. An example of this, in the context of space and sound, is the development of polychoral music in response to the opposing chorister positions in St Mark’s Basilica, which suggests a linkage between auditory and spatial modalities conditioned by a single source [22]. While most studies depend on the mentioned pre-existing semantic relationships to simulate already established links between modalities, few discuss the opportunities of developing synthetic relationships across disparate and yet familiar domains. Historically, architects have adopted such multidisciplinary approaches towards reference material from other fields and modalities to develop new designs. Eisenamans 1987 competition entry for the Biozentrum [21] and Xenakis’ design for the 1958 Philips Pavilion [20] are examples of how genetic code and music were cross-referenced to develop spatial outputs. However, these proposals are dependent on direct translations without broader pattern recognition capabilities. Instead of limiting generative outputs to a preexisting semantic context or adopting rule-based translations, this study explores strategies of applying data pairing techniques to train models on disparate datasets. We propose that by pairing naturally unpaired data, it is possible to establish synthetic connections for working with abstractions systematically. By encoding and decoding information, the role of references is extended to include abstract and complex patterns that are inaccessible to people through traditional means of computation or comprehension. The strategies covered in this paper build on the historic use of cross-disciplinary references in the field of architectural design. Therefore, although our proposed techniques are applicable to a broad range of data types, the study is focused on pairing independent data across music and architectural domains. It is important to note that the research was conducted during 2021-2022, when state-of-the-art model architectures, such as visual transformers or di"usion networks, were in their infancy. The proposed methodologies were appropriate for the time, but may not reflect the current state-of-the-art techniques and high-fidelity results. Regardless, we believe that the work still holds value for the proposed data pairing strategies, the case study and discussion on future improvements. 2Method State-of-the-art multi-modal models such as CLIP [3] and AudioCLIP [2] rely on pre-existing paired data where images or audio are paired with descriptive labels to perform cross-modal translations. For these models, natural language facilitates consistency between the embedding spaces for each data point. However, consistently paired information is often not available when considering alternative and somewhat unlikely cross-modal relationships such as audio and 3D space. Besides acoustic properties, these modalities are not often comparable and lack a semantically meaningful or readily interpretable comProc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 662 Pairing Disparate Datasets for Generative Outputs mon description. Without this, they can not be trained to create a common embedding without prior processing and pairing. To synthesise connections between architecture and music, we propose a pipeline linking unrelated data between architectural models and audio. This data is used to train a CNN model on 12 musical styles and a series of building shapes derived from 6 classical portland stone structures in London. 2.1 Data Representation, Collection and Augmentation For both datasets, we chose to encode spatial and auditory expression into an RGB image format with a 3 x 256 x 256 shaped array. Although our initial ambition was to train the encoder network using 3D architectural information, in our experiments this has proven challenging due to the lack of available curated 3D datasets and issues with non-homologous samples for working with convolutional neural networks. Open source datasets for 2D [5] or 3D CAD information [18,19], are limited in architectural qualities. Popular repositories such as ShapeNet [19] focus on the shapes of everyday objects rather than large-scale spaces or buildings, which limit training models for traditional architectural output. Furthermore, as 3D models do not necessarily translate to consistent homologous tensors, they have to be pre-processed using point clouds or voxelisation techniques. In our experience, these tend to create outputs with substantially more visual noise. We also propose to pre-process audio information and convert it into the same image format. Although 1D mono audio samples are more manageable, the RGB format allows us to keep each data sample homologous for training and pairing purposes. This is also helpful for evaluation as the generated output can be visually compared. The architectural image dataset consists of 3 subsets for plans, sections and elevations. This is proposed in mind of traditional architectural drawing types used to embed 3D information into 2D space, which can also be used for recreating 3D models. To develop the dataset, first, 3D CAD models of 6 Portland Stone buildings in London are subdivided into individual rooms. Then a series of plans, sections and elevation depth maps are extracted from each model. To capture the variety of spatial qualities, sections are cut at regular intervals across each model. This allows to augment the dataset as multiple sections, plans and elevations can be extracted from a single model. The output geometry is then rescaled for consistent inputs and converted into images. For the audio dataset, over 27 hours of music is chosen by the author from 12 genres to ensure variety and diversity between the samples. We used the Librosa library to convert each audio sample into mel spectrograms, which were then processed to match the dimensionality of the building dataset for training consistency. After conversion, each sample was split into multiple 3 x 256 x 256 images to provide homologous samples which roughly correlate to 5 second increments. As seen in Fig. 1 and table 1 the smaller dataset of architectural drawings is not as dense or evenly spread out as the spectrogram dataset. In our early Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 663 R. Markevi!ius Original architectural section dataset Augmented architectural section dataset Fig.1: Architectural section dataset distribution using t-SNE tests, we found the models trained on the initial architectural dataset underperformed when compared to the versions trained on augmented datasets. We therefore trained three StyleGAN2-ADA [14] models for 1,840 steps and used the generated outputs to augment each of the architectural datasets. Although no new original samples are provided in the augmented dataset, we believe the improvements are driven by the adjustments to the embedding positions regarding density and vector distance between samples. Improvements through aligning dataset sample embeddings are further covered in the discussion sections. Table 1: Dataset sample sizes for original and augmented datasets. Dataset Original Augmented Architectural plan dataset samples 284 20,000 Architectural section dataset samples 646 20,000 Architectural elevation dataset samples 1,296 20,000 Mel spectrogram dataset samples 19,069 - 2.2 Feature Extraction and Pairing As each sample consists of 196,608 dimensions, we first encode them into higher level embeddings to improve the computational e#ciency of the pairing process. To meaningfully reduce the dimensionality of the images, each sample is run through a modified large scale VGG-16 model [17]. The final two layers of the model are frozen and replaced with a fully connected feedforward layer outputting 4,000 dimensional embeddings. The large scale pre-trained VGG model Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 664 Pairing Disparate Datasets for Generative Outputs reduces the dimensionality of the inputs with considered compromise to the original samples by utilizing the abstraction from the initial convolutional layers. Due to the broad training set of the original VGG model, we believe the produced embeddings retain most of the initial features. The dimensionality of the generated outputs is further reduced to two dimensions using t-distributed Stochastic Neighbour Embedding (t-SNE) [16]. We propose two approaches for pairing the disparate datasets. First by using Hierarchical Agglomerative Clustering with Ward’s method to pair data based on feature clusters, and secondly by using Self-Organising-Maps (SOM) [13] to pair on a sample basis [1]. Hierarchical Agglomerative Clustering (HAC) To automatically cluster the 2D embeddings, Ward’s method for Hierarchical Agglomerative clustering from the Scikit library is used to partition data into clusters. The 2D t-SNE embeddings allow to make clear distinctions between clusters with di"ering features (see Fig. 2). These are then paired recursively following greedy matching based on shortest distance. This logic is then applied again for the samples within each paired cluster. Any leftover samples without pairs are pruned. Section dataset t-SNE using 9 clusters Section dataset t-SNE using 50 clusters Spectrogram dataset t-SNE using 9 clusters Spectrogram dataset t-SNE using 50 clusters Fig.2: Architectural section and spectrogram dataset t-SNE representations partitioned using Ward’s method for agglomerative clustering with 9 and 50 clusters Self-Organising-Map Overlay (SOM) Each of the datasets is reorganised using Self-Organising-Maps. We used the Somoclu library to reshape each of the 2D embedding sets into matching arrays of the same shape. This allowed us to restructure float information into a more controlled spatial arrangement while preserving the initial relative relationships between the samples. This can be seen through the U-matrix maps of each SOM (see Fig. 3) After a series of tests we derived that a 65 by 65 array produced an optimal density grid. Each sample was then paired based on correlating coordinates of each matrix with any duplicates culled. As with agglomerative clustering, we ended up with 3 paired datasets to test. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 665 R. Markevi!ius Spectrogram dataset SOM u-matrix Section dataset SOM u-matrix Fig.3: Section and Mel spectrogram representations through SOMs. The umatrices on the left highlight the di"erences between the two distributions. 2.3 Model Architecture To perform the cross-modal translation, a Pix2Pix style CNN model [12] is trained on the paired datasets. The model architecture is chosen for its simplicity and ease of training and is implemented using PyTorch and trained on a single RTX-4070-12GB GPU. Samples were separated into training and validation sets using a 90/10 ratio for each dataset. After a series of tests we opted to use a batch size of 32 with 100,000 steps for the architectural models and 70,000 steps for the spectrogram models. Once trained, the supervised models can generate outputs from one sample to another. However, the model architecture does not permit bidirectional translations, meaning that two sets of models have to be trained for bidirectional translation between samples. Table 2: Dataset sample sizes for original and augmented datasets. Methodology Dataset Training pairs Test pairs HAC-Ward-50 Augmented 13,702 1,523 HAC-Ward-9 Augmented 13,436 1,493 HAC-Ward-50 Original 536 60 SOM Augmented 2,621 25 Random Augmented 16,667 1,852 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 666 Pairing Disparate Datasets for Generative Outputs 3Results 3.1 Evaluation Methodology To quantitatively evaluate model performance, we tested generated outputs with LPIPS [10], CMMD and KID evaluation metrics. The LPIPS metric was used to assess output accuracy and was derived through the average score of 500 generated samples compared to ground truth images. In addition, the CMMD [23] and KID [24] metrics were used to assess how well on average the statistical distribution matched between 500 generated outputs and 1000 reference images to review sample diversity and semantic content. All the input, reference and ground truth images were sampled outside the training or test datasets. However, as these techniques rely on pre-trained classifiers using data that is largely dissimilar to our implemented datasets, the results can be inaccurate. We therefore additionally conducted an anonymous survey for a subjective qualitative assessment.120 participants were asked to evaluate visual image clarity (V-clarity), image continuity (V-continuity) and audio clarity (A-clarity) for each pairing methodology. We provided a set of 15 randomly sampled outputs for the visual clarity assessment, 10 generated outputs for the image continuity assessment and 5 sets of 5 second audio for the audio clarity assessment. The image continuity samples were generated using a latent walk sequence of mel spectrogram samples extracted using StyleGAN2-ADA. For audio and visual clarity, all sample sets were generated using identical input data. For the audio samples, the Gri#n-Lim Algorithm (GLA) [7] was used to rebuild the spectrograms into audio. The participants were also provided with reference material for ’very good’ and ’very poor’ samples and asked to evaluate each methodology on an integer scale from 0-10. For a consistent evaluation across quantitative and qualitative metrics, the survey results were inverted by subtracting each score from 10 such that lower scores indicated better performance. We used the Friedman test to conduct whether participants rated the five methodologies di"erently. For this, mean ranks were calculated for each methodology, with lower ranks indicating better perceived performance. Statistical significance was evaluated at ω=0.05. Both the quantitative and qualitative evaluation metrics were applied to the HAC and SOM pairing techniques using the architectural (A) and spectrogram (S) outputs. To test cluster count and data augmentation strategies, we also included separate results for HAC using the augmented dataset with 9 and 50 clusters and the original (OG) dataset with 50 clusters. As a comparative test, we also introduced a random (RND) pairing. Samples for each methodology are available in the appendix. 1Survey evaluation samples are available from the following link: https://drive.google.com/drive/folders/11OXbehUfLQCoPC9xn0-K2pzRSwPQwRP?usp=sharing Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 667 R. Markevi!ius Table 3: Methodology performance across quantitative and qualitative metrics (lower is better). The qualitative measurements (V-clarity, V-continuity and Aclarity) are represented as the mean ranks of each methodology. Methodology LPIPS CMMD KID V-clarity V-continuity A-clarity HAC-50-A 0.1894 0.0205 0.1428 1.500 2.175 - HAC-9-A 0.2119 0.0376 0.2622 2.400 2.200 - HAC-50-A-OG 0.4591 0.3906 2.7497 4.900 4.300 - SOM-A 0.4264 0.2432 1.7143 4.075 3.225 - RND-A 0.5492 0.2095 1.4225 2.125 3.100 - HAC-50-S 0.2541 0.1338 0.8956 - - 2.275 HAC-9-S 0.2490 0.1494 1.0033 - - 3.275 SOM-S 0.2310 0.2144 1.4166 - - 2.000 RND-S 0.4388 0.1885 1.2484 - - 2.450 Spectrograms-to-sections LPIPS Spectrograms-to-sections CMMD Spectrograms-to-sections KID Sections-to-spectrograms LPIPS Sections-to-spectrograms CMMD Sections-to-spectrograms KID Visual clarity mean rank Visual continuity mean rank Audio clarity mean rank Fig.4: Methodology performance across quantitative and qualitative metrics (lower is better). 3.2 Result Evaluation For the qualitative assessment when generating section outputs, HAC with 50 clusters achieved the best performance in all three LPIPS (0.1894), CMMD Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 668 Pairing Disparate Datasets for Generative Outputs (0.0205) and KID (0.1428) metrics. The second-best score in all three categories was also HAC with 9 clusters, which suggests that a di"erence in cluster count has impact on both accuracy and distribution. While SOM-A scored third regarding LPIPS, it performed worse than the HAC-50-A-OG and RND-A regarding the CMMD (0.2432) and KID (1.7143) distribution metrics. For spectrogram pairing, SOM scored best for accuracy with the lowest LPIPS (0.2310). However, the di"erence is marginal as HAC-50-S and HAC-9-S both produced comparable results. Notably, both HAC variants outperformed SOM-S regarding CMMD and KID distribution metrics, suggesting a better overall distribution alignment despite lower accuracy. For the qualitative visual clarity assessment, the Friedman test revealed significant di"erences in participant ratings across all five methodologies (ε2= 66.46, df = 4, p < 0.001). Participants consistently rated HAC-50-A highest (1.50), while HAC-50-A-OG received the lowest ratings (4.90). The original dataset score was also worse than random pairing, suggesting that data augmentation to align dataset distribution substantially improves visual output quality. For visual continuity the Friedman test revealed a similar pattern with significant di"erences across the methodologies (ε2= 25.46, df = 4, p < 0.001) with HAC-50-A scoring best (2.175). Finally, for audio clarity the Friedman test also showcased significant di"erences between methodologies (ε2=11.72,df=4,p < 0.05). In contrary to the visual assessments, SOM-S scored the best (2.000) and HAC-9-S received the lowest rating (3.275). The di"erences between the mean rank of each methodology here are the least significant between the 3 assessments. We attribute the reduced significance between methodologies to artifacts introduced by the Gri#n-Lim Algorithm (GLA) for spectrogram to audio translation. The GLA often results in low reconstruction quality due to the cost function only accounting for consistency rather than characteristics of the target signal. Alternative methodologies for encoding and decoding audio information should be considered in future research. 4Discussion 4.1 Model Evaluation The results show that our proposed data-pairing techniques lead to improvements for generated image and audio quality when compared to a random pairing. This showcases the validity of the discussed data pairing strategies even with modestly sized datasets. Of the proposed techniques, HAC-50 achieved superior performance in almost all evaluation metrics across both section and spectrogram datasets. We believe the superior performance stems from the ability to identify higher-level features common across samples within each cluster, enabling more robust crossmodal translations. As such, the number of unique or similar samples within a cluster directly influences the granularity of features that are translated across modalities. However, this suggests that abstract pairings using fewer and larger Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 669 R. Markevi!ius References 1. Alvarez-Marin, D., Saldana Ochoa, K.: Indexical Cities: Articulating Personal Models of Urban Preference with Geotagged Data. arXiv preprint arXiv:2001.10615 (2020). https://doi.org/10.48550/arXiv.2001.10615 2. Guzhov, A., Raue, F., Hees, J., Dengel, A.: AudioCLIP: Extending CLIP to Image, Text and Audio. (2021). https://doi.org/10.48550/arXiv.2106.13043 3. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2103.00020 (2021). https://doi.org/10.48550/arXiv.2103.00020 4. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M.: Hierarchical Text-Conditional Image Generation with CLIP Latents. (2022) 5. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting. (2021) 6. Google DeepMind: Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Tech. Rep., Google DeepMind, June 17 2025 7. Gri!n, D., Lim, J.: Signal estimation from modified short-time Fourier transform. (1984) 8. Nichol, A., Dhariwal, P.: GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Di"usion Models. (2021) 9. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic Text-to-Image Di"usion Models with Deep Language Understanding. (2022). https://doi.org/10.48550/arXiv.2205.11487 10. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable E"ectiveness of Deep Features as a Perceptual Metric. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068 11. Meta AI.: Llama4: Advancing multimodal intelligence. Meta AI (2024, April 18). https://ai.meta.com/blog/llama-4-multimodal-intelligence/, last accessed 2025/07/01 12. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-Image Translation with Conditional Adversarial Networks. (2017) 13. Kohonen, T.: Self-Organized Formation of Topologically Correct Feature Maps. (1982) 14. Karras, T. et al. Training Generative Adversarial Networks with Limited Data. Preprint at https://doi.org/10.48550/arXiv.2006.06676https://doi.org/10.48550/arXiv.2006.06676 (2020). 15. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Di"usion Models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695. IEEE, 2022. https://doi.org/10.48550/arXiv.2112.10752 16. Maaten, L., Hinton, G.: Visualizing Data using t-SNE. Journal of Machine Learning Research **9**, pp. 2579–2605 (2008) http://jmlr.org/papers/v9/vandermaaten08a.html 17. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. (2014) Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 676 Pairing Disparate Datasets for Generative Outputs 18. Collins, J., Goel, S., Deng, K., Luthra, A., Xu, L., Gundogdu, E., Zhang, X., Yago Vicente, T.F., Dideriksen, T., Arora, H., Guillaumin, M., Malik, J.: ABO: Dataset and Benchmarks for Real-World 3D Object Understanding. arXiv preprint arXiv:2110.06199 (2021). https://doi.org/10.48550/arXiv.2110.06199 19. Chang, A.X., Funkhouser, T.A., Guibas, L.J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. arXiv preprint arXiv:1512.03012 (2015). https://doi.org/10.48550/arXiv.1512.03012 20. Varga, B.A.: Conversations with Iannis Xenakis. Main edn. Faber, London (2003) 21. Lynn, G., Eisenman, P., Gehry, F.O., Yoh, S. (eds.): Archaeology of the Digital: Peter Eisenman, Frank Gehry, Chuck Hoberman, Shoei Yoh. Sternberg Press, Berlin (2013) 22. Matthews, E.-K.: The architectural instrumentalist – exploring spatio-temporal interdependence in the composition of performed music and architectural space. Design Ecologies 6(1), 34–60 (2017). https://doi.org/10.1386/des.6.1.34_1 23. Jayasumana, S. et al. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Preprint at https://doi.org/10.48550/arXiv.2401.09603 (2023). 24. Bi#kowski, M., Sutherland, D. J., Arbel, M., Gretton, A. Demystifying MMD GANs. Preprint at https://doi.org/10.48550/arXiv.1801.01401 (2021). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 677