Generating Abstract Rhythm Streams

Master in Sound and Music Computing Universitat Pompeu Fabra Generating Abstract Rhythm Streams Justin Bosma Supervisor: Sergi Jorda Co-Supervisor: Behzad Haki August 2025 Master in Sound and Music Computing Universitat Pompeu Fabra Generating Abstract Rhythm Streams Justin Bosma Supervisor: Sergi Jorda Co-Supervisor: Behzad Haki August 2025 Contents 1 Introduction 1 1.1 Motivation.................................. 1 1.2 Outline ................................... 2 1.3 Supplementary Material . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Related Works 4 2.1 Architectures ................................ 4 2.2 Transformers ................................ 5 2.2.1 Input Representation of Transformer Model . . . . . . . . . . . . . . . 5 2.2.2 PositionalEncoding ............................ 5 2.3 Variational Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3.1 LossCriteria ................................ 6 2.4 Transformer Based Models for Music Generation . . . . . . . . . . . . . 7 2.5 Non-Tokenized Transformer Inputs and Representation . . . . . . . . . 8 2.6 Utilizing the Latent Space for Generative Purposes . . . . . . . . . . . 8 2.7 RhythmicFeatures ............................. 9 2.8 GrooveTransformer............................. 10 2.8.1 Overview .................................. 10 2.8.2 Architecture................................. 11 2.8.3 Data..................................... 11 2.8.4 Streams/Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.8.5 Groove.................................... 12 3 Design 13 3.1 Streams ................................... 13 3.2 InstrumentOverview............................ 14 3.2.1 Encoder ................................... 15 3.2.2 LatentSpace ................................ 17 3.2.3 Decoder ................................... 18 4 Datasets 20 4.1 Annotated Candombe Recordings . . . . . . . . . . . . . . . . . . . . . 21 4.2 LAKH.................................... 22 4.3 GrooveMIDI................................. 23 4.4 ElBongosero ................................ 25 4.5 TapTamDrum................................ 26 4.6 Dataset Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5 Model 29 5.1 Architecture................................. 29 5.2 DataandTraining ............................. 30 5.3 Validation.................................. 31 5.3.1 Density ................................... 35 5.3.2 Output Stream Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.3.3 Relationships between Feature Values . . . . . . . . . . . . . . . . . . . 38 6 Discussion 41 6.1 Data..................................... 41 6.2 Features................................... 43 7 Conclusion 45 7.1 DataChanges................................ 45 7.2 FeatureChanges .............................. 46 7.3 Future Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 List of Figures 48 Bibliography 50 A First Appendix 53 B Second Appendix 56 Dedication I dedicate this work to my mother, Linnae. Without your constant support, I would never have been able to be the person I am today. I am so grateful to have a mother as wonderful as you. 2Chapter 1. Introduction melody creation, harmonic additions to existing melodies, the generation of rhythms related to the piece of music, and various tools to aid in the mixing and mastering activities. Although these systems have helped in the creative process, they tend to be very specific in their implementation, such as rhythm generation systems with a mapping of their outputs to specific drums like kick, snare, and hi-hat. For our system, currently known as Triple Streams, we aim to create a real-time system that focuses on the creation of abstract rhythmic streams. Our intent is to push the user to think more methodically about how to apply these rhythmic streams to enhance their performance without sacrificing creative choices. To achieve this goal, we pursue three main objectives: 1. Dataset Creation: Build a collection of diverse and interesting rhythmic sequences drawn from both drum-based and non-drum sources, moving beyond one-to-one mappings between instruments and output streams. 2. Model Design: Develop a generative model with real-time control features, enabling performers to manipulate outputs dynamically and treat the system as a playable instrument. 3. Evaluation: Ensure that the system produces diverse, musically meaningful rhythmic streams that relate coherently to the user’s input. 1.2 Outline •Chapter 2: Related Works – Overview of prior work that informs our architecture, data representation, and control features. We examine relevant machine learning models, rhythmic feature representations, and the GrooveTransformer system that serves as our foundation. •Chapter 3: Instrument Design – High-level overview of the system’s design, partitioned into encoder, latent space, and decoder components. We also describe the system’s inputs, outputs, and interactive controls. 1.3. Supplementary Material 3 •Chapter 4: Data Curation – Description of datasets used for training and validation, preprocessing methods, challenges encountered, and the rationale behind design choices. •Chapter 5: Model Overview and Validation – Presentation of the model’s architecture, training process, and evaluation of rhythmic outputs, including the effect of control features on system performance. •Chapter 6: Discussion – Reflection on the system’s current state, identifying strengths, limitations, and possible improvements in data, control features, and output quality. •Chapter 7: Conclusion – Summary of key findings, open challenges, and planned future work, including refinements to datasets, control mechanisms, and deployment. 1.3 Supplementary Material Throughout this thesis, we will use supplementary material to provide additional insights and explanations to the text. All download links, processed data, figures, code, and other resources used in this thesis can be found in the following github repository: https://github.com/justinbosma/TripleStreams Chapter 2 Related Works Over the past decade, numerous contributions have been made to the intersection of audio, music, and machine learning, and each contribution can be seen as a stepping stone to our current work. However, in this section we will focus on the most related works to our current research, and how they directly impact our implementation. We will begin with a brief overview of the Machine Learning models employed in our system. Following this, we will examine some of the more recent generative models that have had a strong influence on our decisions. Finally, we will give an overview of the prior work of Behzad Haki, specifically the GrooveTransformer, allowing us to show where our current research develops from. 2.1 Architectures Due to the sequential nature of symbolic notation, and specifically rhythm, we employ a transformer model for the modeling of temporal dependencies. To allow us to create variations on rhythms, we utilize a variational autoencoder(VAE) to encode they rhythms into a latent space, where we can interpolate between similar rhythms for slight variance from the generations based on the input. Below, we will give an overview of both architectures. 4 2.2. Transformers 5 2.2 Transformers First described by Ashish Vaswani in their 2017 paper "Attention is All You Need", the transformer is a deep learning model that has become fundamental to the area of Natural Language Processing (NLP) [3]. This model differs from Recurrent Neural Networks (RNN) in that it allows parallel processing of input sequences, making training and inference more efficient. The two important aspects of transformer models are positional encoding and selfattention. In the example of text, positional encoding assigns a number to each word, rather than looking at its position in the sentence. This provides information about the token, allowing the model to consider sequential information. With selfattention, each word is assigned a weight and is compared against the other words in parallel. This allows the model to "learn" the grammar based on how words are used in everyday life. 2.2.1 Input Representation of Transformer Model At the bottom most encoder, the input is first broken down into tokens, which would be words and subwords in NLP. Each token is assigned a unique index from the model’s vocabulary. After assigning the unique index, each token is converted into a dense embedding vector of size V×d, where V is the vocabulary size and d is the embedding dimension. 2.2.2 Positional Encoding To keep track of the order of the sequence, Transformers use a positional encoding to maintain the information about the relative or absolute position of the token. This can be achieved by using the following equation: PE(pos,2i)= sin pos/100002i/dmodel  6Chapter 2. Related Works where pos is the position and iis the dimension. Each dimension iof the positional encoding is related to a sinusoid [3]. 2.3 Variational Autoencoders Variational autoencoders are self-supervised deep learning models that allow us to isolate important information, known as latent variables, via an encoder. The latent variables are then used to decode our input. The input is not encoded directly, but is encoded in a probabilistic distribution, from which the numerical representation is sampled [4]. One of the most important aspects of Variational Autoencoders is the latent space, which is the space consisting of all of the latent variables from our dataset. The data is compressed into a smaller dimensional latent space, where only the meaningful information extracted from the autoencoder is represented. This encoding of information into a latent space allows us not only to recreate our original input but also to generate new data. 2.3.1 Loss Criteria For training the VAE, we use the reconstruction loss to reconstruct the input from the encoded space. This can be written formally as: Lrecon(x,ˆx) = difference(x,ˆx) where we attempt to minimize the loss between difference(x,ˆx). The reconstruction loss will differ depending on the type of data that we are using, for example, with tokenized language, we could calculate cross-entropy loss between the predicted and original sequences [5]. Lrecon(x,ˆx) = −X t xtlog(ˆxt) Because we are interested in generating new samples from our input data, we cannot rely only on a loss function. We need to use another regularization term to ensure 2.4. Transformer Based Models for Music Generation 7 that we can sample from anywhere in the latent space between our original data points. To achieve this, we use the Kullback-Leibler (KL) Divergence, where we ensure the latent space follows a standard normal distribution. The KL Divergence is formally defined as: LKL =DKL N(µ(x), σ2(x)) ∥ N(0,1) From here, our total loss can be defined as follows: Ltotal =Lrecon +LKL Having a latent space that is continuous, where nearby points yield similar samples when decoded, and complete, where all the points in the latent space should have meaningful content when decoded, is an important aspect of our implementation. This allows us to interpolate between two data points in our space. 2.4 Transformer Based Models for Music Generation The Music Transformer, developed by Huang et al in 2019 incorporates a relative positional attention mechanism, which allows for the creation of songs up to one minute in length. They show that their new implementation outperforms LSTMbased models like Performance RNN in maintaining motifs, repetition, and structure over long spans. It is also capable of continuing an existing melody or harmonizing with a given melody [6]. The Pop Music Transformer by Huang et al. is an extension of the Music Transformer that is more capable of capturing the rhythmic structure of pop music [7]. The implementation focuses on enhancing the representation of input/output sequences. This was made possible by the use of REMI (REvamped MIDI derived events) tokenization, explicitly encoding bars, beat positions, tempo changes, and chords alongside typical note events. 8Chapter 2. Related Works 2.5 Non-Tokenized Transformer Inputs and Representation In their paper preceding the implementation of GrooVAE, Gillick et al. suggest the use of an enhanced piano roll representation of drums [8]. This representation uses a binary piano roll relative to a fixed grid, setting onsets to ’1’ and rests to’0’. They also incorporate the use of velocity and offsets, micro-time deviations in relation with the grid, into their representation. This allows for a simple representation of drums that can be used as input for their system. In his work on GrooveTransformer, Behzad Haki uses this HVO (hit, velocity, offset) representation for the input and output of the system. 2.6 Utilizing the Latent Space for Generative Purposes Over the past decade, there have been numerous real-time models for the use in music and sound generation. We will look at some of these that relate to our project and show how these models can help in our research. MusicVAE is a recurrent variational auto-encoder model developed for creation of symbolic music of varying lengths. The authors note that the application of VAEs to sequential data has been limited, and they struggle to model sequences with long-term structure. In an attempt to solve this issue, they employ the use of a hierarchical decoder, which outputs embeddings for subsequences and then uses the embeddings to generate new subsequences [9]. This extension of MusicVAE can generate loops by sampling points in the latent space. It also allows one to interpolate between two loops, by moving between points in the latent space, which allows one to mix between the loops for more musical control. In his GrooveTransformer implementation, Behzhad Haki utilizes this technique of interpolating between two loops in the latent space, and is something we will include in our new implementation. GLSR-VAE is another approach to loop generation using recurrent VAEs. It employs 2.7. Rhythmic Features 9 regularization techniques to improve the distributions of the embeddings in the latent space. This allows for a smoother transition between loops in the space, giving more control and similarity, while still allowing interesting variation [10]. An important application in this work to our own is the observation that a dimension in the latent space can be associated with the density of a generated sequence, and moving in one direction can increase the density of the sequence. This will be utilized when we implement our control features for our final system. Generative Adversarial Networks (GANS) have also been utilized in the creation of musical loops. One example of this is MidiNet (Yang et al.), where a convolutional GAN is used to create musical sequences [11]. They propose a conditional mechanism to use available prior knowledge, allowing the model to generate melodies from scratch, following input via chord sequences, or by conditioning on the prior melodies. Other approaches to creating rhythmic spaces like the one mentioned above include RhythmVAE by Tokui and R-VAE by Vigliensoni et al. [12] [13]. Both of these use VAEs to map drum patterns into 2-dimensional latent space encodings. While moving through this space, the user can generate random patterns, slightly modify patterns, and interpolate between various patterns to create interesting rhythmic sequences. 2.7 Rhythmic Features For our implementation of the Triple Streams rhythm generating system, we wish to create various control features to allow the performer to have more control over the generated streams. Gomez-Marín et al. have recorded a general set of rhythmic features, describing various similarity metrics for 1-bar drum patterns, which have been collected and tested according to human ratings [14]. The features mentioned in the paper include density of onsets for the total rhythm, as well as looking at the density of onsets in low-, mid-, and high-frequency bins. They also define a means of looking at the syncopation on both a global level and the individual frequency bins 10 Chapter 2. Related Works as well. We can utilize this idea of looking at the features of individual frequency bins and apply it to our individual streams. From their work, we will implement features based on onset density, rhythm similarity, and syncopation. Furthermore, we can use the features described in their paper for validation of our generated rhythmic streams, allowing us to ensure that we have a diverse set of rhythms after decoding. 2.8 GrooveTransformer The GrooveTransformer, a controllable drum accompaniment generation system, was developed by Behzad Haki, with input from Catalan musician Raül Refree. It is a real-time rhythm generation system that outputs drum rhythms based on a user’s input, intended for use in live performance. Our new implementation will be heavily based on this work, and we look to build on this work for our implementation of the Triple Streams system. Now, we will give a brief overview of the system as a whole, the model architecture, data sources, and the representation of the streams. 2.8.1 Overview The current implementation of GrooveTransformer was created by Behzad Haki for their doctoral research at Universitat Pompeu Fabra. The system is based on years of experience working in the area of rhythm generation and representation, and in its current form, it is a redesign based on the feedback from the musician Raül Refree [5]. The focus of the redesign is to allow for a balance between performance and composition, which is attained by addressing the dualism of autonomy and control of the generated rhythmic streams. Initially, the system could only be updated by changing the input groove or changing the sampling parameters. This was modified to give the performer more control over the system, by the inclusion of two additional predetermined patterns that allowed the user to interpolate between the two additional patterns and the input groove. The addition of a VAE into the model’s architecture allowed for the interpolation between these three rhythms, enabling the 2.8. GrooveTransformer 11 performer to modify the output in real-time [5]. In addition, two new controls were included in the new system implementation. The first of these, five individual mute controls, allow for the muting of the instrument groupings "kick, snare, hats, toms, cymbals. These are enabled by learning an embedding with the same dimension as the latent space and added to the latent vector before decoding. The other control, a genre mute/selector, is created by including genre features in the training process by adding the information to the encoder, and similarly to the instrument mutes/genre information sent to the decoder. 2.8.2 Architecture The model architecture for the current implementation of the GrooveTransformer consists of a Transformer VAE , where a 2-bar rhythmic pattern is encoded into a latent distribution, and is decoded into a 9-voice drum pattern. The model consists of three parallel decoders, one for hits (onsets), velocity, and offsets (microtimings). 2.8.3 Data GrooveMIDI was chosen as the dataset for the GrooveTransformer system, as it has a large amount of multi-drum MIDI files that can be used for training the model. In addition, the MIDI files have an associated genre tag, allowing for the creation of genre filtering in the system. Due to the genre-imbalance of the dataset, additional private data was used to balance the dataset [5]. 2.8.4 Streams/Representation GrooveTransformer uses an HVO matrix to represent the hits, velocities, and offsets for both the input and output rhythms. The matrices are of size T x M, where T is the time steps in the representation in sixteenth notes, and M is the number of instruments. In the final GrooveTransformer implementation, these HVO matrices will be of size 32 X 27. We get 32 time steps from 2-bar rhythm sequences composed of sixteenth notes in 4/4 timing, and 27 from multiplying the 9 instruments by 3 for the hits, velocities, and offsets. For our TripleStreams implementation, we will use 18 Chapter 3. Design Expanding on the ideas in the GrooveTransformer, we want to allow the user to move beyond the stored latent variables in a linear direction. This gives the user a simple way to explore the space, hopefully yielding interesting results while still maintaining some relation to the two rhythms. Looking at figure 6, we can see the user’s input rhythm encoded in the top node labeled ZG, and the two saved data points labeled ZAand ZB. The diagram on the right of this figure shows how we can interpolate between the two data points, but also allow the user to move in a linear direction beyond the path between the two points. Figure 6: Representation of the Latent Space for Real-Time Interpolation 3.2.3 Decoder Our final subsection is the decoder. In this region, rhythms that have been encoded in the latent space are decoded into a flattened stream. We then expand this flattened stream into our three outputs based on the density values we have selected. Density After decoding the latent variable, we receive our flattened output of the three streams. To keep the design as simple as possible, we decided to include only one set of controls on this end. We designate these controls as ’Density’, which controls the number of onsets in each expanded stream. The number of onsets in each stream is bounded by the total number of hits in the flattened output. If the flattened output 3.2. Instrument Overview 19 has 12 onsets, the maximum number of onsets each unflattened stream can have is 12. This allows the user to quickly modify the individual streams according to their needs during a performance. To calculate the density for each stream, we employ the Jaccard Similarity between each individual expanded stream and the original flattened output. We define this formally as Streami∩Flattened Streami∪Flattened Looking at figure 7 we can see the expansion of the flattened output into the three streams based on the density feature control values. Setting our density feature control to 70% for the top stream allows us to have 7 of the 10 onsets found in the flattened output. Both velocity and offset values will be retained from the flattened stream. Figure 7: Expanding the output into the three streams based on the Density Feature Control Values Chapter 4 Datasets A central focus of this research is exploring different ways to extract rhythmic information from diverse sources, moving beyond a drum-centric perspective in which each generated stream corresponds to a specific drum. Instead, we investigate more abstract methods for constructing rhythmic streams. This includes drawing on datasets that are not strictly percussive as well as developing strategies for partitioning and recombining data to reconstruct rhythms. During the data curation process, we applied several criteria. First, the dataset needed to carry a suitable license; those under the Creative Commons Attribution 4.0 International (CC BY 4.0) license were considered acceptable. Second, because our method requires accurate tempo and time signature information to compute offset values from MIDI files, only datasets that retained this metadata were usable. This restriction excluded some otherwise valuable collections. Third, we prioritized datasets with a broad range of velocities and offsets in order to provide richer training data for the model. Based on these criteria, we selected a set of annotated Candombe recordings, the LAKH MIDI dataset, GrooveMIDI, El Bongosero, and TapTamDrum. The following sections detail our rationale for choosing these datasets, the specific processing steps used to reduce them into four streams, and the challenges encountered. We then describe the conversion of the curated material into the HVO rep20 4.1. Annotated Candombe Recordings 21 resentation used for training, along with the construction of a new dataset created specifically for this project. All data manipulation was implemented in Python and is available in the project’s GitHub repository. For MIDI-based datasets, we relied extensively on PrettyMIDI for reading, analyzing, modifying, and writing MIDI files [15]. All datasets used fall under the Creative Commons Attribution 4.0 International license, making them suitable for our purposes. 4.1 Annotated Candombe Recordings The dataset of Candombe recordings was created by Luis Jure, Martín Rocamora, Simone Tarsitani, and Martin Clayton. It consists of multiple live recordings of Candombe artists recorded in a studio in Uruguay in 2018. Specifically, one player on the piano drum, one on the chico, and two on repique [16]. The dataset also contains annotations of onsets and velocities. Offset values were not included, but calculated by Satyajeet Prabhu and Anmol Mishra. The offset values are only for the tracks with one repique, so we will omit the recordings with two repique players. In the tradition of Candombe music, each drum has a specific role in the performance. The piano, which is the largest of the drums, has the lowest frequency and provides a melodic foundation for the other drums. The repique is the second largest of the three drums and is used to add variation and improvisation to the performance. The ’llamada’, or call and response, is performed on these drums and is central to Candombe. Finally, the chico is the smallest of the three and is used to provide the stable rhythm for the performance. The Candombe rhythm comes from the interaction of these three drums and their functions, creating a very complex rhythmic structure. Though we have stated we wish to move away from mapping one drum to one output stream, we found it was best to have each drum represent one stream, our fourth stream being a flattened combination of the three drums. This allows for the complex rhythmic structure to be maintained in our model training. 22 Chapter 4. Datasets The data preprocessing code for our Candombe dataset can be found in the ’candombe.py’ file in our repository. 4.2 LAKH The LAKH dataset consists of 176,581 unique MIDI files, including 45,129 files that have been matched to songs in the Million Song Database [17]. For our purposes, the LAKH dataset offers an assortment of instruments that we can use for creating various rhythmic streams, and allowing different instruments to fill the role of the ’Groove’, e.g. using bass guitar as the main groove, and using the piano, electric guitar, and drum instruments as the related streams. Due to the size of the LAKH dataset, we need to be precise with how we choose our combinations of instruments. First, to condense the large amount of instruments into their families, like guitar, electric guitar, lead guitar into ’Guitar’, we match on all the associated MIDI program numbers and merge these into one instrument with the name we will be using. We then look at the potential instruments to use and choose those that we assume to be better suited for our task. For example, a combination like "Piano", "Guitar", "Drums", and "Bass" seems like a classic example that would have a good underlying rhythmic structure that we can utilize. However, certain things like "Strings" or "Synth Pads", tend to have long sustained notes and may not have the most interesting onset and micro-timing information for our research, so these are excluded. Our original subset of instruments to use consists of "Piano", "Percussion", "Guitar", "Bass", "Brass", "Ethnic", "Percussive", "Sound Effects", "Drums", and "Synth Effects". To find which combinations of these instruments are most represented in our dataset, we create a small subset of 4,200 MIDI files, plot a histogram of the top ten combinations of four of the instruments we have selected, and choose these for creating our rhythmic streams. In figure 8 we can see that "Bass", "Drums", "Guitar", "Piano", "Brass", "Percussion", and "Percussive" have the highest numbers, so we will use these for our rhythmic stream creation. Because the LAKH MIDI dataset consists of entire songs, it was decided to only use 4.3. GrooveMIDI 23 Figure 8: Top 10 instrument combinations from LAKH dataset. a portion of each MIDI file. There are many 2-bar loops that are repeated and these become redundant in our training. To accomplish this, we sort the HVO sequences for each MIDI file by the number of total hits. We then select the second half of the sorted list and take eight evenly spread HVO sequences. This will also help reduce instances where there are little to no onsets in the HVO sequences. The code for our LAKH MIDI preprocessing, as well as the code for selecting the eight HVO sequences can be found in lakh_midi.py and lmd_best_eight.py. 4.3 GrooveMIDI The GrooveMidi dataset contains more than 13.6 hours of aligned MIDI of human performed drumming and offers us a great deal of material to use for training [18]. Behzad Haki’s GrooveTransformer system was trained on this dataset to create its output drum streams. For our purposes, we are looking for other ways we can merge these various drum tracks into three to four streams. In "Drum Rhythm Spaces" (Gómez-Marín et al. 2020), it is noted that humans perceive the roles of various drums in three frequency bands, i.e., low, mid, and high. We use this as a way to separate the MIDI recordings into three separate streams, using a flattened version of the three as the groove [19]. Our hope is that this will maintain some of the ’functions’ of the drums, e.g., the kick drum as a pulse in western music, the snare drum to reinforce the meter in rock, etc. The three partitions we create include 24 Chapter 4. Datasets the low partition, including kick drum and low tom, the mid partition, including snare drums, mid toms, and high toms, and the high partition, including hi-hats, ride cymbals, and crash cymbals. For our second grouping, we partition the recordings into velocity bins. Loudness and its MIDI representation, velocity, can be used to emphasize certain portions of the rhythm. These emphasized moments could be accents to reinforce the meter, or interesting events in a musical piece. Our intention was to partition these emphasized portions, where one stream would consist of onsets with high velocity that would stand out against the other sounds, one stream would be very subtle and could be used for minor changes, and something in between. From a performance perspective, these could be very useful when mapped to parameters such as synth timbre and envelopes to control effects like delay. The third grouping that we use is rooted in the traditions of western rock, pop, and electronic music. We focus on four instrument families, kick drum, snare, toms, and hi-hats, and create new MIDI files based on these groupings. This allows us to abstract the roles of the instruments slightly, not mapping each individual drum to a stream, but focusing on how they are used in these styles of music. Our last two groupings were chosen as potentially interesting groupings that would also yield more sparse streams. One of these is a collection of the cymbal instruments, where one stream is the open hi-hat, one is the closed hi-hat, one is the crash cymbal, and the last is the ride cymbal. Certain styles of music, such as jazz, have a tendency to incorporate cymbals into the rhythm, and we assume this could be an interesting grouping. In a similar fashion, we make a grouping out of the three toms and the ride cymbal. GrooveMIDI is a large dataset, and these five groupings allow us to diversify the HVO sequences that we create from it. After processing our dataset, we have 82,687 2-bar rhythms split into five different groups. We can see in figure 9 the velocity, functional, and pitch (frequency) groupings, each representing roughly a quarter of the dataset, which is expected. The other two have less representation because they 4.4. El Bongosero 25 Figure 9: Percentages of each GrooveMIDI grouping for total dataset. are drum specific, and certain recordings may not have all the individual drums to create the grouping. The code for our preprocessing of the GrooveMIDI dataset can be found in the file named groove_midi.py. 4.4 El Bongosero El Bongosero is a large-scale symbolic dataset created by Behzad Haki et al. consisting of 6,035 crowd-sourced improvised drum performances by 3,184 participants of varying levels of experience [20]. The participants were asked to select a genre for a backing track from the GrooveMIDI dataset and improvise on a set of digital bongos over a 2-bar loop. Once finished, they could listen to their recording and overdub additional hits. The participants were then asked to specify their level of experience out of a maximum of 5 points, as well as rate their performance out of 5 points. For our dataset, we choose to use the events from the left hand, right hand, a combination of both left and right hands, and the flattened GrooveMIDI track. The inclusion of the ’both hands’ event allows us to extract interesting rhythms that are 26 Chapter 4. Datasets directly related to the two other rhythmic streams by sharing overlapping onsets, and can aid in the training of the model in regards to Rhythmic Similarity, Accent Similarity, and the Density features. To better curate our dataset, we include only sessions with an expert ranking of 4 or higher and a user rating of 3 or higher. With our restrictions in place, the El Bongosero dataset is reduced to 2243 total recorded sessions. Of these remaining recorded sessions, 1917 contain events where the left hand plays, 1863 where the right hand plays, and 1221 where both hands play simultaneously. This can be seen in figure 10. Given the number of hits for each of our selections, we feel that our choice of events from the El Bongosero dataset is justified and will add interesting diversity to our final dataset for model training. Figure 10: Recorded Session Counts for Left Hand, Right Hand, Both Hands, and total sessions for our filtered El Bongosero dataset. The preprocessing code for our El Bongosero dataset can be found in the file bongosero.py. 4.5 TapTamDrum TapTamDrum is another symbolic dataset created by Behzad Haki et al. where four expert drummers are tasked with improvising on two drum pads over 2-bar loops chosen from GrooveMIDI [21]. The sessions last approximately one hour, and each 4.6. Dataset Pre-processing 27 expert drummer plays a variety of genres. After each session, the performers are asked to rate their session out of five points. We follow a similar method with the El Bongosero dataset, using the groupings left hand, right hand, both hands, and the flattened GrooveMIDI track. To add variety to our training data, we create two datasets from TapTamDrum, allowing the ’both hands’ category to be when both hands play simultaneously (Intersection) or when at least one hand is playing (Union). Additionally, we restrict our dataset to entries that have a user rating of 4 or higher. After restricting our dataset, we have 1116 sequences in each dataset, totaling 2232 TapTamDrum HVO sequences we can use for training. Our code for the TapTamDrum dataset can be found in the file tap_tam_drum.py. 4.6 Dataset Pre-processing The datasets described above are represented in various ways. LAKH and GrooveMIDI are both MIDI datasets, and require the additional usage of PrettyMIDI to open, merge the instruments, and rewrite to MIDI files. The Candombe annotations were stored in CSV files, and only required the use of Pandas for extracting the data and saving into HVO representations. Because they included the onsets, velocities, and offsets, we only needed to transfer this information to new HVO objects. Both El Bongosero and TapTamDrum were already in HVO representation, only requiring the transfer of the hit, velocity, and offset values into new HVO objects. For the data preprocessing task, the MIDI datasets, LAKH and GrooveMIDI, took the longest to create. First, we had to group the various instruments into their larger family, such as electric piano and acoustic piano into ’Piano’. After we designated the groupings, we saved each individual instrument family for each file, to be loaded into HVO sequences via the HVO MIDI loader. It was observed that additional notes were written to the newly created MIDI files, which was caused by very short durations in the original MIDI file that were smaller than the time intervals supported by PrettyMIDI. To overcome this, we wrote a small method to remove any 34 Chapter 5. Model the total velocities changed. This implies that the model struggles to create streams with changing accents for values between the maximum and minimum. Looking at figure 16 we see that our Accent Similarity feature is not working as well as we had expected. We would expect this to have a similar shape as the graph from the Rhythmic Similarity, growing monotonically. For the original model, there is some growth, but the rate of change is very slow until it reaches the feature values of 8 and 9, where it increases much more rapidly. In the flex model, we see no change in our Hamming Distance and we can assume that reducing the number of tokens is not a valid way to improve our model. The variance also increases between values of [−7,7]. This shows us our current implementation is not a reliable control in it’s current state. (a) Hamming Distance for various Accent Similarity values using original model (b) Hamming Distance for various Accent Similarity values using flex model Figure 16: Hamming Distance for various Accent Similarity values of original and flex models Another issue can be observed in figure 17. When looking at stream 1 and stream 2 for the maximum distance value, we can see there are many values that are less than those of stream 3 (the darker shading represents higher velocities). This implies that there is a relationship between the density values and the Accent Similarity, and we shall explore these unintended relationships later in this section. 5.3. Validation 35 (a) Accent Similarity Minimum (b) Accent Similarity Half (c) Accent Similarity Max Figure 17: Output Streams for Three Accent Similarity values 5.3.1 Density In Section 3.2.3, we explain how the density feature is calculated for each of the output streams. Our intent was to have a control that could dictate the number of onsets in each stream, relative to the flattened output. For many of the outputs, our expectations are met by the number of onsets in each stream. We can see in figure 18 setting stream 1 to two, stream 2 to five, and stream 3 to nine gives us a stream with very few onsets, and the others add more based upon the values specified. The same holds for the image on the right. Stream 1 and stream 3 are set to the maximum value and stream 2 is set to one. Both images are using a Rhythmic Similarity distance set to zero, so should mirror the groove displayed on the bottom. (a) Density Values set to 2, 5, and 9 out of 9. (b) Density Values set to 9, 1, and 9 out of 9. Figure 18: Various Density Values for Output Streams However, when we decide to maximize all the values, we can see there are some issues in the decoding. The pattern on the left of figure 19 has a density value of seven for stream 1, nine for stream 2, and nine for stream 3, out of a maximum of nine. All other controls are set to zero. The image on the right has all density values for the streams set to nine, the maximum value. We would assume that setting each 36 Chapter 5. Model stream’s density value to the maximum in this scenario would contain all the onsets of the input groove, and each stream would be the same as the other. But, both stream 1 and stream 3 have less onsets than stream 2, as well as the input. This may be an issue with training, or with how we calculate the onsets on the decoding side, and more investigation needs to happen. (a) Density Values set to 7, 9, and 9 out of 9. (b) Density Values set to 9, 9, and 9 out of 9. Figure 19: Issues with Densities set to Maximum Values 5.3.2 Output Stream Quality To judge the general quality of the output streams, we will look at three criteria. The first is that the onsets create an interesting pattern. By interesting, we mean a pattern that has rhythmic qualities, but has variations in the positions of the onsets. An example of uninteresting results would be having many patterns generated for various grooves, but all the generated onsets appear on the down beat, or where the same pattern is repeated each measure. Our second criterion is to look at the velocities of the generated streams. We expect to see variation in the velocities, accenting certain onsets, and giving the rhythm multidimensionality. For our third criteria, we are concerned with the offset values of the generated onsets. This shows our model is capable of recreating these offsets, and can output rhythms that consist of triplets that aren’t aligned to the grid. In figure 20 we have four different piano rolls displaying the output for four different input grooves. The values of the feature controls have been randomly selected. The piano rolls for figures a, b, and c all show interesting variation for the onsets. We 5.3. Validation 37 have instances where little to no repetition happens in the 2-bar loop, such as for figures a and b. In figure c, we see he pattern repeats itself for each bar, but this may be an issue with the input, which also includes three repeating notes for each bar. In figure d, we see the least amount of diversity. The second bar is a copy of the first for each of the three streams. Furthermore, Stream 2 contains onsets on all but one position in each bar. Our assumption is the input groove’s repetitive pattern, and a high density value for stream 2, is what causes this lack of diversity. Looking at the velocity values for these four examples creates some concern. For each of the streams in each of the patterns, velocity values do not exceed a value of 0.7. The input grooves all contain various velocities between minimum and maximum values, so we would expect to see more varying values for our output streams. This has been a common occurrence in all of our attempts, and not specific to these four examples. Changing the Accent Similarity value will cause velocity values to be raised, but this differs from our expectations. The Accent Similarity should shift the velocity values. If we have a large difference, onsets that have a large velocity in the input should have smaller velocities in the output streams, and onsets that have small velocities in the input stream should have larger velocity values in the output streams. (a) (b) (c) (d) Figure 20: Four Examples of Generated Streams for Four Different Inputs 38 Chapter 5. Model Finally, when we look at the offset values for out outputs, we see much less variation than we had hoped for. The only example that has offset values less than or greater than 0 is figure b. The other three examples only contain onsets snapped to the grid. Each of the input grooves have varying levels of offsets, so we cannot say that this is an issue with the inputs. Furthermore, after many attempts, we still find that most of our outputs contain little to no variation in the offset values. Our assumption is that this is caused by a lack of offset diversity in the LAKH MIDI training set and something we will discuss later in this paper. 5.3.3 Relationships between Feature Values One of the unintended relationships we have noticed in our system is between the Rhythmic Similarity control and the Accent Similarity control. In figure 21 we look at how changing the Rhythmic Similarity distance affects the Accent Similarity and vice versa. (a) Original Model - Rhythmic Similarity held constant while increasing Accent Similarity (b) Flex Model - Rhythmic Similarity held constant while increasing Accent Similarity Figure 21: Issues with Densities set to Maximum Values If we set the Accent Similarity distance to the maximum value, and our Rhythmic Similarity distance to something low, we see we have onsets in the same positions as the input groove, and velocity values that are higher than the lower velocities in the input. However, as we increase the Rhythmic Similarity distance we notice the velocity values began to decrease in our output. This can be seen in figure 22. The image on the left shows our output with a maximum distance for the Accent Similarity and a distance of 1/3 the maximum for Rhythmic Similarity. Our velocity values have a maximum of 0.754 and a minimum of 0.180. On the right we can see 5.3. Validation 39 what happens when we raise our Rhythmic Similarity to the maximum values. Here our velocities have a maximum value of 0.528 and a minimum value of 0.088. We would expect to maintain the same level of velocity for our output, and to have high velocity when the same position in the input either contains an onset with a low velocity or no onset at all. This issue may be due to the similar ways we calculate these two features and we shall discuss this later in the paper. (a) Density Values set to 2, 5, and 9 out of 9. (b) Density Values set to 9, 1, and 9 out of 9. Figure 22: Velocity values decreasing as we raise the Rhythmic Similarity Distance A second unexpected relationship was found between the Rhythmic Similarity feature and the Density features for each of the output streams. In figure 23 we see three different scenarios when we have a very active input with many onsets. (a) Rhythmic Similarity distance value of 8 out of 32 (b) Rhythmic Similarity distance value of 19 out of 32 (c) Rhythmic Similarity distance value of 32 out of 32 Figure 23: Relationship between Rhythmic Similarity and Density for active input When the Rhythmic Similarity distance value is low, we allow our output onsets to happen at the same position as the input onsets. If we set our density values to larger amounts, such as the maximum, we see very active outputs. This is expected behavior. More density implies more onsets. However, as we increase the amount of distance for our Rhythmic Similarity feature, we see the onsets in the output streams decreasing. Knowing how the features are calculated, this makes perfect sense. The Rhythmic Similarity measures the distance between the input and the 40 Chapter 5. Model flattened output. A maximum distance between the two would only allow onsets in our output to occur at locations where there are no onsets in the input. For an input containing many onsets, a maximum distance for the Rhythmic Similarity would contain very few onsets. Because of this, the density for each of the streams would be bounded by the number of onsets in the flattened output stream. If there are only four locations for our onsets to occur, a maximum density value would allow four onsets in that stream. However, this can be very confusing from a user’s perspective. In most situations, increasing the density will increase the number of onsets in the output, and it would be safe to assume that many users will have this assumption. Chapter 6 Discussion 6.1 Data In our discussion of the model we have seen that our system does not quite meet our expectations. Specifically, we have noticed issues around the rhythmic structure of our outputs being too repetitive, unintended relationships between our features for controlling the rhythm outputs, and a lack of variation in our training data. In this section, we will look at the specific problems of our current model and how we can improve it in the future. One of the issues we raised is the lack of variation in our output rhythms. As we have seen, we often have outputs that do not contain much diversity for velocity values or for offset values. We have also noticed a tendency to create outputs that have repetitive patterns in the sequence and lack variation in the placement of the onsets. One of the biggest factors creating this issue is the datasets we have used, specifically the LAKH MIDI dataset. Although the dataset allowed us interesting groupings from the multiple instruments used, the patterns in the MIDI files are lacking in the variation of velocity and offsets that we were hoping to find in our output. In figure 24 we have three examples of the LAKH MIDI training data. 41 42 Chapter 6. Discussion (a) Example 1 of LAKH MIDI Training Data (b) Example 2 of LAKH MIDI Training Data (c) Example 3 of LAKH MIDI Training Data Figure 24: Three Examples of LAKH MIDI Training Data We can see there is very little variation with the velocity values, offset values, and the onset positions. Comparing these piano rolls with those generated from our other datasets 25, we can find much more diversity in our hit, velocity, and offset values. (a) Example of Candombe Training Data (b) Example of El Bongosero Training Data (c) Example of TapTamDrum Training Data Figure 25: Training Data from Candombe, El Bongosero, and TapTamDrum Datasets One of the issues with the LAKH dataset is that it contains full songs. When we split these full songs into 2-bar HVO sequences, we find that many of the sequences do not have the qualities we have mentioned, such as diversity in the velocities or offsets. Initially, we filtered the full songs from the original LAKH MIDI dataset by the number of velocity changes, but never did this for the 2-bar split HVO sequences. To rectify this, we can remove 2-bar HVO sequences that do not have a certain number of velocity or offset changes. In figure 26 we can see the number of velocity changes for a small subset of our LAKH MIDI 2-bar splits used for training. Out of 267713 HVO sequences, 174496 have only one velocity value. This will have a huge impact on training our model. However, if we restrict the HVO sequences to ones with six or more velocity changes, we still have 17,002 remaining sequences. Our full LAKH MIDI dataset contains sixty partitions of the 2-bar HVO sequences, which 6.2. Features 43 would provide more than enough data to meet our criteria to train our model. Figure 26: Velocity Changes in 2-bar splits for a Subset of LAKH MIDI 6.2 Features Another area of concern that we have mentioned is the relation between the control features of our system. Specifically, the relation between the Rhythmic Similarity and the Accent Similarity controls and the relation between the Rhythmic Similarity and the Density controls. Our intention with the Accent Similarity control was to allow the user to inject a feeling of syncopation into the output, in relation to the input groove. However, we discovered that it is difficult to find a reasonable way to calculate and implement syncopation to be used as a control in our system. Furthermore, the similarities between the way we calculate the Rhythmic Similarity and Accent Similarity controls has made it very difficult to isolate how they impact the output of the system. For future work, we will remove the Accent Similarity, and focus on ensuring the Rhythmic Similarity works exactly as we intend. Once we are certain that there are no issues with our feature on the encoding side, we can begin to add additional features and be more aware of their impact on the output. Another realization is the need for the generation of the output to be invariant of the feature controls. The controls should affect how the onsets are distributed and arranged, but not the number of onsets in the flattened output. Moving the Rhyth- Bibliography [1] Jiang, H. H. et al. AI art and its impact on artists. In Rossi, F., Das, S., Davis, J., Firth-Butterfield, K. & John, A. (eds.) Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2023, Montréal, QC, Canada, August 8-10, 2023, 363–374 (ACM, 2023). URL https://doi.org/10.1145/ 3600211.3604681. [2] Nugroho, Y. Y. T. & Manggala, P. P. M. D. The use of ai in creating music compositions: A case study on suno application. In Proceedings of the 7th Celt International Conference (CIC 2024), 177–189 (Atlantis Press, 2024). URL https://doi.org/10.2991/978-2-38476-348-1_13. [3] Vaswani, A. et al. Attention is all you need (2023). URL https://arxiv.org/ abs/1706.03762.1706.03762. [4] Kingma, D. P. & Welling, M. Auto-encoding variational bayes (2022). URL https://arxiv.org/abs/1312.6114.1312.6114. [5] Haki, B. Design, development, and deployment of real-time drum accompaniment systems (2025). [6] Huang, C. A. et al. An improved relative self-attention mechanism for transformer with application to music generation. CoRR abs/1809.04281 (2018). URL http://arxiv.org/abs/1809.04281.1809.04281. [7] Huang, Y. & Yang, Y. Pop music transformer: Generating music with rhythm and harmony. CoRR abs/2002.00212 (2020). URL https://arxiv.org/ abs/2002.00212.2002.00212. 50 BIBLIOGRAPHY 51 [8] Gillick, J., Roberts, A., Engel, J. H., Eck, D. & Bamman, D. Learning to groove with inverse sequence transformations. CoRR abs/1905.06118 (2019). URL http://arxiv.org/abs/1905.06118.1905.06118. [9] Roberts, A., Engel, J. H., Raffel, C., Hawthorne, C. & Eck, D. A hierarchical latent vector model for learning long-term structure in music. In Dy, J. G. & Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, vol. 80 of Proceedings of Machine Learning Research, 4361–4370 (PMLR, 2018). URL http://proceedings.mlr.press/v80/roberts18a.html. [10] Hadjeres, G., Nielsen, F. & Pachet, F. GLSR-VAE: geodesic latent space regularization for variational autoencoder architectures. In 2017 IEEE Symposium Series on Computational Intelligence, SSCI 2017, Honolulu, HI, USA, November 27 - Dec. 1, 2017, 1–7 (IEEE, 2017). URL https://doi.org/10.1109/ SSCI.2017.8280895. [11] Yang, L., Chou, S. & Yang, Y. Midinet: A convolutional generative adversarial network for symbolic-domain music generation using 1d and 2d conditions. CoRR abs/1703.10847 (2017). URL http://arxiv.org/abs/1703.10847. 1703.10847. [12] Tokui, N. Towards democratizing music production with ai-design of variational autoencoder-based rhythm generator as a DAW plugin. CoRR abs/2004.01525 (2020). URL https://arxiv.org/abs/2004.01525.2004. 01525. [13] Vigliensoni, M. L. M. E., G. & Fiebrink, R. R-vae: Live latent space drum rhythm generation from minimal-size datasets. Journal of Creative Music Systems 1(1) (2022). URL https://doi.org/10.5920/jcms.902. [14] Gómez-Marín, D., Jordà, S. & Herrera, P. Drum rhythm spaces: From global models to style-specific maps. In Aramaki, M., Davies, M. E. P., KronlandMartinet, R. & Ystad, S. (eds.) Music Technology with Swing - 13th International Symposium, CMMR 2017, Matosinhos, Portugal, September 25-28, 2017, 52 BIBLIOGRAPHY Revised Selected Papers, vol. 11265 of Lecture Notes in Computer Science, 123– 134 (Springer, 2017). URL https://doi.org/10.1007/978-3-030-01692-0_ 9. [15] Raffel, C. & Ellis, D. P. W. Intuitive analysis, creation and manipulation of MIDI data with pretty_midi. In Proceedings of the 15th International Conference on Music Information Retrieval Late Breaking and Demo Papers (2014). [16] Jure, L., Rocamora, M., Tarsitani, S. & Clayton, M. Iemp uruguayan candombe (2025). URL osf.io/wfx7k. [17] Raffel, C. Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching. Ph.D. thesis, Columbia University, USA (2016). URL https://doi.org/10.7916/D8N58MHV. [18] Gillick, J., Roberts, A., Engel, J., Eck, D. & Bamman, D. Learning to groove with inverse sequence transformations. In International Conference on Machine Learning (ICML) (2019). [19] Gómez-Marín, D., Jordà, S. & Herrera, P. Drum rhythm spaces: From polyphonic similarity to generative maps. Journal of New Music Research 49, 438–456 (2020). [20] Evans, N., Haki, B., Gómez-Marín, D. & Jordà, S. El bongosero: A crowdsourced symbolic dataset of improvised hand percussion rhythms paired with drum patterns. In Kaneshiro, B. et al. (eds.) Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, 540–546 (2024). URL https://doi.org/10.5281/zenodo.14877393. [21] Haki, B., Kotowski, B., Lee, C. L. I. & Jordà, S. Taptamdrum: A dataset for dualized drum patterns. In Sarti, A. et al. (eds.) Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, 114–120 (2023). URL https://doi.org/10. 5281/zenodo.10265237. Appendix A First Appendix During our discussion around the validation of the model, we chose not to discuss the KL Divergence loss during training and validation. In figure 27 we can see that our original model is properly reducing the loss during training and validation. (a) KL Divergence Training Loss (b) KL Divergence Test Loss Figure 27: Kullback-Leibler Divergence for Training and Validation In the Related Works section we introduced the transformer model. Being that this thesis is focused on the design aspect of our system, and transformer models have been used substantially in the past few years, we chose to not go too into detail with our description. For more information on transformer models, one can read Attention is All You Need (Vaswani et al. 2017) [3] We can see the architecture of the transformer model in figure 28. 53 54 Appendix A. First Appendix Figure 28: Transformer model architecture [3] We also chose to not go too into detail with the Variation AutoEncoder model for the same reasons as we gave for the transformer model. We include a figure of the VAE architecture here 29. In his doctoral dissertation, Behzhad Haki gives an in-depth explanation of the VAE architecture and we refer the reader to his paper [5]. 55 Figure 29: Variational Autoencoder Architecture [5] Appendix B Second Appendix 56