Fine-Grained MIDI Expression Transcription from Wind and String Instrument Audio via Sim2Real Transfer Learning

Fine-Grained MIDI Expression Transcription from Wind and String Instrument Audio via Sim2Real Transfer Learning Yifan Xie1, Zixun Guo1,andMathieuBarthet 1,2 1Centre for Digital Music, Queen Mary University of London, London, England 2Aix-Marseille University CNRS PRISM, Marseille, France {y.xie, zixun.guo, m.barthet}@qmul.ac.uk Abstract. While MIDI velocity estimation in piano music transcription has been widely studied, similar work for other instruments remains underexplored. Unlike piano MIDI velocity, which provides note-level volume modulation, MIDI Expression (CC11) provides continuous volume modulation across a note’s duration, requiring finer temporal resolution. This paper addresses the task of estimating MIDI CC11 values from wind and string instrument audio recordings. To explore suitable estimation methods, we first investigate the numerical relationship between MIDI CC11 and audio Root Mean Square (RMS) energy. Motivated by the analysis results of the MIDI CC11–RMS relationship, we compare three estimation approaches: linear, quadratic, and BiLSTM-based deep learning. We adopt a Simulation-to-Reality (Sim2Real) strategy, training models on synthetic audio rendered from randomized MIDI CC11 curves and evaluating on real performance recordings. Unlike approaches requiring manually labeled data, ours relies entirely on synthetic training, avoiding the need for expert annotation. Experiments on violin, viola, flute, and trumpet demonstrate the e!ectiveness of the Sim2Real approach, with the deep learning model achieving the best performance. Using the deep learning model, we generate a MIDI dataset enriched with fine-grained MIDI CC11 annotations, which can be used for future expressive music analysis, modeling, or generation. All transcribed data is available online [3]. Keywords: Automatic music transcription ·MIDI Expression (CC11) ·Expressive music performance ·Deep learning 1Introduction MIDI velocity estimation has been widely explored in automatic music transcription, particularly for piano performance [9,11,15,16,18]. MIDI velocity, which All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 570 Y. Xie et al. reflects the speed at which a key of a piano is pressed, is a dynamic control parameter in MIDI. However, beyond the piano, there is limited research on estimating MIDI control parameters a!ecting dynamics for other instruments, such as winds and strings. A fundamental di!erence between piano and wind/string instruments lies in how each note is played. In piano, the volume of a note is determined at its onset by MIDI velocity and decays naturally over time. In contrast, wind and string instruments sustain notes through continuous bowing or blowing, enabling performers to modulate volume dynamically throughout the whole duration of the note. This temporal continuity requires more fine-grained modeling. While MIDI velocity provides note-level volume modulation, MIDI CC11 is a parameter that operates at a more fine-grained level, allowing for continuous modulation of volume throughout the whole duration of a note (see [4] for more details). Both parameters contribute to the shaping of musical dynamics, and their estimation can be viewed as part of the broader task of dynamics transcription. In this paper, we address the MIDI CC11 estimation problem for strings and winds. We focus on monophonic music and define our task as mapping a sequence of audio frames to a corresponding MIDI CC11 sequence, as illustrated in Figure 1. This figure shows a 0.5 s audio clip divided into 10 ms frames, each associated with a MIDI CC11 value. Also, this task assumes the availability of paired datasets consisting of human performance audio and its corresponding MIDI representation, which includes annotations for pitch, note onset, and o!set, but is devoid of MIDI CC11 information. Fig. 1. MIDI CC11 values are predicted from each audio frame To explore which prediction methods are suitable, we first analyze the numerical relationship between MIDI CC11 and RMS energy (throughout the rest of the paper, we refer to RMS as the RMS energy of an audio frame). MIDI CC11 controls volume modulation, which directly influences perceived loudness, while RMS energy is widely used as a proxy for loudness in audio analysis. Under fixed synthesis conditions—such as constant pitch, fixed frame position, and static MIDI CC11 context—the relationship can be well approximated by linear or quadratic functions. However, when introducing variations in pitch, frame position, and surrounding MIDI CC11 values, the relationship becomes more complex. Based on this analysis, we experimentally compare three estimation approaches: linear fitting, quadratic fitting, and deep learning. In this paper, we employ a Sim2Real [10,13,14] approach to transcribe MIDI CC11 from audio recordings. Unlike methods that rely on manually labeled datasets—which require substantial e!ort from expert musicians—our approach does not require any manually annotated controller information. Specifically, we Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 571 MIDI Expression Transcription via Sim2Real Transfer generate synthetic training data by applying randomly generated MIDI CC11 sequences to MIDI sequences and rendering the corresponding audio using synthesizers. Our experiments are conducted on four di!erent instruments: violin, viola, flute, and trumpet. Despite being trained exclusively on simulated random data, the model demonstrates strong generalization to real performance recordings. Among the three modeling methods evaluated, the deep learning model achieves the best performance under this Sim2Real setting. Also, based on the deep learning method, we obtain a MIDI dataset enriched with fine-grained MIDI CC11 annotations, which can be used for future expressive music generation. In summary, our contributions are as follows: 1. We tackle a new but related problem to MIDI velocity estimation in piano performance: predicting MIDI CC11 from wind and string instrument audio recordings. Unlike MIDI velocity, MIDI CC11 operates at a finer temporal resolution. To the best of our knowledge, this is the first study about performing dynamic transcription into MIDI for string and wind instruments. 2. We provide an in-depth analysis of the MIDI CC11-RMS relationship, considering various factors such as pitch, frame position, and contextual MIDI CC11 changes from adjacent frames. 3. We propose a Sim2Real transfer learning approach to transcribe MIDI CC11 directly from audio without requiring any manually labeled data. By using this Sim2Real transfer learning strategy, we generate synthetic training data, achieving strong generalization and transcription performance on real-world recordings. 4. By transcribing audio performances with our deep learning method, we obtain a MIDI dataset enriched with fine-grained MIDI CC11 annotations, which can be used for future expressive music modeling or generation. The same transcription method can be applied to other recordings to create larger MIDI CC11-labeled datasets for downstream tasks. 2RelatedWork Dynamics in music have been studied extensively as part of expressive performance modeling. Early work [20] shows how performers vary dynamics in ways that reflect expressive intentions, laying the groundwork for computational models of expressive deviations. [21] provides an authoritative review of computational methods that extract expressive parameters such as timing and dynamics from performance recordings. Moving beyond general expressive dynamics modeling, researchers have turned attention to MIDI velocity estimation specifically from piano performances. There are generally two approaches for estimating MIDI velocity in piano performance transcription. The first approach is integrated into a comprehensive music transcription task, where onset, o!set, pitch, and MIDI velocity are transcribed simultaneously [16,18]. The second approach isolates MIDI velocity estimation by leveraging known pitch and timing information (onset and o!set) to aid in Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 572 Y. Xie et al. the transcription process [9,11,15]. In this paper, we adopt the second approach by transcribing MIDI CC11 information using known onset, o!set, and pitch information. Some music generation studies have explored dynamic generation for strings and winds. For example, MIDI-DDSP [22] generates dynamics for these instruments but is limited to two cases: crescendo and decrescendo. This approach simplifies dynamic variations, which are often more nuanced in expressive performances. Similarly, NotePerformer [5], a commercial software, generates dynamics for strings and winds but only supports a limited set of dynamics at the note level, such as pp, p, f, !, crescendo, and accent. While e!ective for basic dynamic changes, this approach lacks the detailed and continuous expressive control observed in human-performed audio, where dynamics are shaped with much greater nuance and variability. In contrast, our approach provides finer temporal resolution by estimating MIDI CC11 values. Besides, a recent study [19] introduces a MIDI dataset with fine-level dynamics where modulation curves are manually created by composers to reflect expressive intent. While this provides valuable insight, such annotation requires extensive manual e!ort and musical expertise. In contrast, our work directly transcribes MIDI CC11 from audio using a training approach that leverages synthetic data, thereby eliminating the need for human-labeled annotations. Another di!erence is that their work focuses on MIDI CC1 modulation, whereas we target MIDI CC11, which is more directly associated with volume control in expressive performance for most synthesizers. 3AnalyzingtheMIDICC11-RMSRelationship Table 1. Experimental setting for MIDI CC11-RMS relationship analysis (with Current referring to changes applied to the current frame and Others to other frames) MIDI CC11 (current) Pitch (current) Frame Index (current) MIDI CC11 (others) A0–127 Fixed Fixed Fixed B0–127 Variable Fixed Fixed C0–127 Fixed Variable Fixed D0–127 Fixed Fixed Variable In this section, we analyze the numerical relationship between MIDI CC11 and RMS. Inspired by Dannenberg [8], who observed a quadratic relationship between MIDI velocity and RMS at the note-level, we investigate the MIDI CC11-RMS relationship at the frame level using a 10 ms frame granularity. While [8] focused on isolated notes with fixed pitches, our analysis, summarized in Table 1, expands on this by considering additional factors. Specifically, we first examine how RMS changes with varying MIDI CC11 values for a given frame while keeping pitch, frame position, and the MIDI CC11 values of neighboring frames constant. We then explore how each of these factors individually Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 573 MIDI Expression Transcription via Sim2Real Transfer a!ects the MIDI CC11-RMS relationship. These analyses are conducted across four di!erent instruments including violin, viola, flute, and trumpet. We evaluate the results using two representative synthesizers: FluidSynth [2] and BBC Symphony Orchestra Discover from Spitfire Audio [1]. FluidSynth is chosen for its widespread use in open-source music applications and its seamless integration with Python programming environments, while the BBC Symphony Orchestra represents high-quality orchestral sampling. 3.1 MIDI CC11-RMS Relationship under Fixed Settings We first examine the MIDI CC11-RMS relationship under the simplest conditions: fixed pitch, fixed frame position, and static MIDI CC11 values in adjacent frames. Specifically, for each instrument, we create a 5-second note with a MIDI pitch of 60 (C4), select a 10 ms frame starting at 2 s, and record the average RMS values as the MIDI CC11 value of that frame is varied from 0 to 127. The FluidSynth results shown in Figure 2 show a consistently quadratic pattern across instruments, with R2>0.9998,thoughthecoe"cientsvaryfordi!erentinstruments. In contrast, the BBC Symphony Orchestra results shown in Figure 3 follow a linear relationship, also with R2>0.9998, but with instrument-specific slopes. Fig. 2. MIDI CC11-RMS Relationship under fixed settings with FluidSynth Fig. 3. MIDI CC11-RMS Relationship under fixed settings with BBC Symphony Orchestra from Spitfire Audio 3.2 MIDI CC11-RMS Relationship Influenced by Pitch Next, we examine how pitch a!ects the MIDI CC11–RMS relationship. Using FluidSynth, we render a 5-second note, vary MIDI CC11 from 0 to 127 for a 10 ms frame at 2 s, and sweep the pitch from 60 to 100. Figure 4 shows that the relationship remains quadratic across pitches on the viola, with R2>0.9998, though the coe"cients vary with pitch in a non-linear and non-monotonic way (Figure 5). This trend holds across all four instruments. We conduct a similar experiment using the BBC Symphony Orchestra plugin. The MIDI CC11–RMS relationship remains approximately linear across pitches, but the slope varies non-linearly (The figures are not included here due to page limitations). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 574 Y. Xie et al. Fig. 4. MIDI CC11-RMS relationship across di!erent pitches on the viola using FluidSynth Fig. 5. The x2coe"cient changes with the change of MIDI pitch using FluidSynth 3.3 MIDI CC11-RMS Relationship Influenced by Frame Position We then analyze how di!erent frame positions within a note influence the MIDI CC11-RMS relationship. To do this, first, for FluidSynth, we render a 5-second note and vary the starting position of a 10 ms frame from 0 s to 3 s in increments of 10 ms. The results for several di!erent positions on the flute are shown in Figure 6. The quadratic relationship remains consistent across all positions. Additionally, Figure 7 shows the changes in the x2coe"cient, which exhibit a complex pattern that is di"cult to approximate with basic functional forms. Similarly, we perform the same analysis on a 5-second note using the BBC Symphony Orchestra from Spitfire Audio. The results also exhibit a change pattern that cannot be captured by simple functions (The figures are not included here due to page limitations). Fig. 6. MIDI CC11-RMS relationship in di!erent frame positions on the flute using FluidSynth Fig. 7. The x2coe"cient changes with the change of frame position using FluidSynth 3.4 MIDI CC11-RMS Relationship Influenced by Time-Varying MIDI CC11 Additionally, we found that changes in MIDI CC11 values from both preceding and subsequent frames influence the RMS of the current frame. To analyze this, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 575 MIDI Expression Transcription via Sim2Real Transfer first, for FluidSynth, we render a 6-second note and compute the average RMS over a 10 ms window starting at 2 seconds. Figure 8 shows the results when varying the MIDI CC11 values of the preceding 200 frames for the violin. While the quadratic relationship is maintained, a regular trend in the changes is observed. However, in real musical performances, MIDI CC11 values in preceding frames vary more unpredictably and dynamically, leading to a more complex relationship. Figure 9 illustrates the MIDI CC11-RMS relationship when varying the MIDI CC11 values of the subsequent frames. Although the quadratic pattern persists, the trend lines exhibit more complex changes in their coe"- cients. We conduct a similar experiment using the BBC Symphony Orchestra from Spitfire Audio by rendering a 6-second note. Figures 10 and 11 show how changing the MIDI CC11 values of the previous 200 frames a!ects the RMS values of the 203rd and 204th frames, respectively. These results indicate that changes in MIDI CC11 influence not only the current frame but also multiple adjacent frames. The cumulative e!ect of these influences contributes to the complex relationship between MIDI CC11 and RMS. Fig. 8. MIDI CC11-RMS relationship when changing the MIDI CC11 from previous frames using FluidSynth Fig. 9. MIDI CC11-RMS relationship when changing the MIDI CC11 from subsequent frames using FluidSynth Fig. 10. CC11–RMS Relationship (RMS at Frame 203) with CC11 from Previous Frames using BBC Symphony Orchestra Fig. 11. MIDI CC11-RMS relationship (RMS at Frame 204) when changing the MIDI CC11 from previous frames with BBC Symphony Orchestra Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 576 Y. Xie et al. 4TranscriptionMethodsUsingSim2RealTransfer Learning From Section 3, we observe that while MIDI CC11 and RMS exhibit a quadratic or linear relationship under fixed settings, the relationship becomes more complex when influenced by various factors. Based on the analysis, we experimentally compare three modeling approaches: linear fitting, quadratic fitting, and deep learning. We employ a Sim2Real approach to transcribe MIDI CC11 from audio recordings. Specifically, we generate synthetic training data by applying randomly generated MIDI CC11 curves to MIDI sequences and rendering the corresponding audio using virtual instruments. We then implement three modeling approaches on this synthetic data: linear fitting, quadratic fitting, and a Bi-LSTM model [12], which leverages sequential learning capabilities to e!ectively capture the complex temporal dependencies present in MIDI CC11-RMS relationships. 4.1 Training Data We generate the training data by creating MIDI CC11 sequences based on predefined rules (described below), apply them to MIDI files with pitch and timing, and render the audio using virtual instruments. This process enables us to generate a large volume of data. During training, the input and output are the reverse of the data generation process: audio serves as the input, and the MIDI CC11 sequence is the output. Specifically, the predefined rules for generating a MIDI CC11 sequence are created by initializing with a random value between 0 and 127, and subsequent values are generated by adding a delta sampled from a Gaussian distribution with a mean of 0 to ensure smooth transitions. A penalty prevents values from exceeding the 0-127 range. This method is designed to simulate the smooth dynamic change of many musical performances, although it does not account for the compositional context. 4.2 Quadratic and Linear Fitting We first experiment with two simple fitting methods: linear and quadratic fitting. Each audio sequence is divided into multiple audio frames, and each frame is treated as a data point consisting of three elements: MIDI pitch, RMS, and MIDI CC11. Suppose there are mdistinct MIDI pitch values in the dataset. For each pitch, we aggregate all data points with that pitch and fit a linear or quadratic function that maps RMS to MIDI CC11. During inference, the MIDI pitch of a frame determines which function to use, and the RMS value is then passed through the corresponding model to predict the MIDI CC11 value. 4.3 BiLSTM-based Deep Learning Method We first introduce the input and output features of the deep learning model. For the input, we use a sequence of audio frames [frame1,frame2, ..., framen], where Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 577 MIDI Expression Transcription via Sim2Real Transfer each frame is represented by a two-dimensional vector [MIDI pitch,RMS]T.To minimize the influence of timbre di!erences between the synthesized training data and real instrument test data, we avoid features like log mel spectrograms. The model predicts a sequence [CC111,CC112, ..., CC11n]corresponding to each input frame. Since MIDI CC11, which is quantized between 0 and 127, represents acontinuousrangeofphysicallymeaningfulcontrol,wetreatthisasaregression task. During training, each CC11nis a floating-point value. During prediction, the final floating-point values are rounded to obtain integer MIDI CC11 values. Next, we present the model architecture. This task involves time-dependent predictions, with a one-to-one correspondence between input and output time steps. We use a Bi-LSTM model, as each frame is influenced by both preceding and following frames as discussed in Section 3.4. 5 Experiments 5.1 Experimental Setups We use the URMP dataset [7,17], a multi-track dataset featuring 14 common orchestral instruments, with separate audio tracks and corresponding onset, o!set, and pitch annotations for each note. In this work, we focus on four instruments: violin, viola, flute, and trumpet, which contain 34, 13, 18, and 22 audio files in the dataset, respectively. For transcription, we use the FluidSynth synthesizer due to its seamless integration with Python, making it convenient for experimental implementation. Additionally, we utilize a 10 ms frame granularity for precise temporal resolution. Next, we introduce the configuration of the deep learning model. The BiLSTM model has an input dimension of 2 (MIDI pitch and RMS features) and an output dimension of 1 (predicted MIDI CC11 value). It consists of 2 bidirectional LSTM layers with a hidden dimension of 128, followed by a multilayer perceptron with a linear layer (2 →hidden_dim,hidden_dim),ReLUactivation,andafinal linear layer (hidden_dim,1). Dropout with a probability of 0.3 is applied to each LSTM layer and the first linear layer. For training the deep learning model, we use Mean Squared Error (MSE) as the loss function and Adam as the optimizer, with a learning rate of 1e-4. For each instrument, we synthesize approximately 3,000 pieces using the MIDI files from the URMP dataset and MIDI CC11 sequences generated randomly based on predefined rules described in Section 4.1. The data is split into training, validation, and test sets with a ratio of 8:1:1, using a batch size of 32. Training, conducted on an NVIDIA RTX A5000, was stopped if validation loss did not improve for 40 epochs, and the model with the lowest validation loss was retained. To evaluate the methods, as mentioned in Section 1, we compare the RMS di!erences between the rendered and real performed music audio. Since RMS values are typically small, we convert them to decibels (dB) for comparison. For each frame f,thedBdi!erenceisdefinedinEq.(1),andourevaluationmetric is the mean absolute dB di!erence in Eq. (2). Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 578