Real-Time Gesture Classification via Multi-Modal Sensor Data for Intuitive Performance Mapping

Real-Time Gesture Classification via Multi-Modal Sensor Data for Intuitive Performance Mapping Evan O’Donnell1and Atau Tanaka1[0000000325211296] Goldsmiths, University of London, London SE14 6PP, UK [email protected], a.tanak[email protected] Abstract. We present an approach to real-time continuous gesture recognition via accessible classification and regression tools and its application in an embodied musical performance workflow. By combining electromyogram (EMG) muscle signals with gyroscope and accelerometer data from a wireless inertial measurement unit (IMU) system, we attained well-rounded descriptors of ongoing gestural arm movements. By training on this data via multilayer perceptron classification and outputting confidence ratings in place of predicted classes, we successfully detected five pre-chosen gesture types in real time and interpolated smoothly between their associated audio clips. Integrating this model into an interactive performance system let us harness these confidence ratings as overlapping influences on a musical arrangement, expanding on embodied musical associations via intuitive aesthetic mappings. Our results demonstrate the feasibility of continuous gesture recognition with the current generation of accessible machine learning tools, extending prior research into new use cases while enabling exploration and application of embodied expressive associations. Keywords: Gesture detection ·body sensing ·performance systems. 1Introduction Gesture-sound mapping is a common technique in digital musical interaction design, often as a means of incorporating embodied musical expression into electronic music practice. Machine learning (ML) holds promise for facilitating these connections, reducing the amount of direct programming required and allowing for subjective, bespoke mapping decisions. However, mapping continuous movements to continuous audio output with ML poses multiple challenges, including appropriate feature selection, adapting to expressive variation, and smooth interpolation between gesture types [21]. Several strategies have emerged over the past two decades, but many are no longer supported within common creative coding environments or require advanced programming skills to implement, All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 937 E. O’Donnell and A. Tanaka while others limit fine-tuning of training procedures. A new generation of accessible ML tools paired with current inertial measurement unit (IMU) and muscle electromyogram (EMG) capture solutions suggest fresh workflows for machine learning of gesture and its integration into creative practice. Our work shares some goals with previous sound tracing research, seeking approaches to gesture-sound pairing that move beyond the indicative or symbolic (e.g. raising a hand to trigger playback) to the descriptive or intuitive (e.g. arm movements that trace the shape of a phrase) [22]. However, training on these intuitive associations poses additional difficulties, such as accurately identifying these for individual users and capturing data in a useful format [8]. Such gesture-sound mappings also pose challenges in performance practice, including generating audio for expressive variations or unrecognized gestures [22]. This paper presents a solution combining intuitive gesture-sound mappings, flexible audio output, and lightweight technical implementation via contemporary, accessible machine learning tools. We use neural network classification and regression via multimodal sensor data to train on pre-determined gesture types, outputting confidence ratings for each class in real time to influence the playback of their associated sounds. This approach embraces the guesswork of the model itself to smoothly interpolate between audio outputs, while enabling the broader mapping of gesture categories within an interactive performance system based on aesthetic associations. In the process, we demonstrate the creative affordances of gesture classification with the current generation of ML tools when embedded in a creative coding environment. 2RelatedResearch Identifying meaningful movement-sound relationships is a central part of designing interactive performance systems. Mainsbridge discusses the need to understand the body’s physical and creative potential when learning to perform with sensor-based instruments [11]. Caramiaux et al. examined movement affordances associated with sounds by participants and examined the nuances of expressive gesture in context via muscle sensors [3]. Visi et al. documented participant’s movements in reaction to sound via musical instruments and wearable sensors to inform action-sound mapping [20]. Sound tracing, a research strategy which measures spontaneous movements in relation to sound, dates back to Becking and Truslit in early 20th century and has been explored more recently by Godøy, et al. [9] and Zbyszyński et al. The latter used sound tracing exercises to map participants’ gestures to a multi-dimensional timbre space via the Myo armband and rapidmax library [22]. Françoise and Bevilacqua harnessed a similar approach in their mapping-by-demonstration workflows, correlating ongoing gestural movements with audio responses [8]. The use of body signals in music dates back to at least the 1960s with Alvin Lucier and David Rosenboom’s use of electroencephalography (EEG) [16]. In the late 1980s Ben Knapp and Hugh Lusted introduced the BioMuse, harnessing EMG signals for sonic interaction, and the second author, Atau Tanaka, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 938 Real-Time Gesture Classification for Performance performed the first concert piece for the device in 1991 [15]. Chris Van Raalte and Ed Severinghaus introduced the BodySynth during the same period, used by Laurie Anderson and Pamela Z in live performance. More recently, Donnarumma developed and performed with the open-sourced, mechanomyogrambased Xth Sense [16]. While EMG offers many expressive affordances, it lacks spatial movement descriptors. In the early 1990s, Laetitia Sonami introduced the Lady’s Glove, controlling sound by sensing body movement with magnetic sensors and accelerometers [14]. In 2002, the second author worked with Ben Knapp to combine EMG with gyroscope readings for multimodal body movement sensing, blending these approaches [17]. The introduction of the Myo armband by Thalmic labs in the 2010s made multimodal sensing widely available [12] and spurred further research until it left the market. At present, there are several EMG sensing solutions available, including the EAVI board which interfaces with the Max environment via the BBDMI library [5,6]. One experimental version of this device also includes an onboard accelerometer. Gyroscopes and accelerometers are available more broadly in an array of consumer devices. Machine learning of gesture also has a long history, with various strategies for distinguishing continuous movement types and mapping them to associated sounds. Wekinator and the related rapidmax library allow for quick prototyping of movement-based interactions, including the use of regression to interpolate between static poses and dynamic time warping (DTW) to classify gestures by comparing them to a series of values over time [7,16]. Gesture Follower (GF) uses probability to identify ongoing gestures with higher tolerance for variation and noise, while Gesture Variation Follower (GVF) tracks differences in the performance of trained gesture types for expressive variation [2,3]. The XMM library allows gesture-to-sound mapping with relatively few training examples, including mapping the time-based evolution of gestures and audio clips to one another [8]. More recently, the Fluid Corpus Manipulation (FluCoMa) library has offered a new range of accessible machine learning tools [19]. Among its chief advantages are smoother integration into environments like Max and Pure Data, greater ease of use for artists with low to moderate coding knowledge, and ahighcapacityforcustomization.Inthispaper,weprovideapractice-based example applying this library alongside the current generation of EMG and IMU tools for real-time gesture recognition, demonstrating a flexible, lightweight, and accessible approach to designing meaningful gesture-sound mappings for an individual performer. 3 Methodology and Implementation 3.1 Data Capture We captured multimodal sensor input in Max, combining a glove-based Mugic IMU unit sending raw gyroscope (x y z) and accelerometer (x y z) readings via OSC over WiFi [10] with EMG readings from the bicep brachii (on the upper arm) and flexor carpi radialis (forearm) via the EAVI board over Bluetooth LE (BLE) to the BBDMI Max library [5,6]. We chose this combination of modalities Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 939 E. O’Donnell and A. Tanaka Fig. 1. Sensor placements for data capture and model testing: A glove-based IMU on the back of the right hand, EMG sensor placements on the bicep and forearm, and an EAVI board strapped to the forearm. for its multifaceted description of arm and hand movement, merging movement descriptors with measurements of effort and intensity. All IMU readings were scaled to a range of -1. to 1., while EMG data was passed through a Bayesian filter and scaled between 0. and 1. All readings were then concatenated into an eight point vector (gyroscope x y z, accelerometer x y z, EMG 1 and 2), sampled every 10 milliseconds and recorded into a Max coll object, indexed by ms clock readings to enable playback at the same rate. Pre-recording this raw data permitted experimentation with feature selection, data smoothing and curation during training without sacrificing performance subtleties or time-specific information. We documented five predetermined gesture types via this workflow (detailed in the following section), capturing at least 60 seconds of readings for each class. This entailed 30 to 40 performance examples for shorter movements and five to 10 examples for longer, repetitive movements, yielding a final data set of approximately 37,000 raw data points. 3.2 Gesture Classes In earlier work by author one, we looked at correspondences between gesture and rhythmic phrase [13], building on Constanzo’s improvisation analysis system [4]. The first author recorded video, audio, and multimodal sensor data from live performances, improvising gestures with one arm while table-tapping rhythmic patterns with the other. He returned to analyze these for trends he found expressively interesting and gesture-sound correspondences useful in training. Based on these prior analyses, we chose five gesture-sound categories which represented some of the most common tendencies in author one’s improvisational language and were distinguishable via our chosen sensor placements. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 940 Real-Time Gesture Classification for Performance Fig. 2. Visualization of one second of multimodal raw sensor data (gyroscope, accelerometer and EMG) for two gesture classes: “circulation” and “stretching”. –Agitation: Quick side-to-side arm movements with the arm held to the front and wrist turned sideways, representing rapid, staccato rhythmic patterns. –Circulation: Rotating the wrist in a steady circular motion while holding the arm out straight, corresponding to a rolling pattern or purring sound. –Idling: Holding the arm forward and steady in the air with minimal movement, to indicate silence or a more static holding pattern. –Landing: A quick downward motion and recoil with the arm along with a wrist snap, indicating firm punctuation within a phrase (such as a heavy downbeat or “thudding” sound). –Stretching: Moving the arm sideways away from the body while holding strong muscle tension, corresponding to a series of tense-sounding clicks (as if delaying the arrival of the next phrase). The first author practiced and standardized these gesture categories for performance consistency, then captured data for each class using the procedure detailed above. 3.3 Machine Learning Workflow and Training To prepare raw data for training, we played each recorded example for each gesture class back in real time, redividing the eight-point vectors into individual sensor readings. After experimenting with several data processing approaches, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 941 E. O’Donnell and A. Tanaka MLPClassifier MLPRegressor Class 1 Confidence Ratings Gyroscope x y z Accelerometer x y z EMG 1 2 Sensor Data Capture 16-Point Vector Transfer Output Class 2 Class 3 Class 4 Class 5 Machine Learning Workflow Overview Data Processing Fig. 3. Overview of the machine learning workflow: Multi-modal sensor data is sent to the classifier for training. The trained model is transferred to a regressor to output confidence ratings for each class for new sensor inputs. we chose to take a running average of the past five values for each sensor data stream, while also sampling these averages every 50 ms and taking the difference between the current and previous reading. We recombined these averages and differences to a 16-point vector for training. This gave us descriptors for both momentary and time-dependent variations within a diverse enough set of sensor values to distinguish each class. To create our training data set, we sampled these 16-point vectors every 50 ms during playback and assigned a unique identifier to each vector, while creating aparallellabelsetassigningeachidentifiertoagestureclass.Forclassification labels, we used the names given to each gesture type in section 3.2, reflecting the shapes of the movements and their sonic associations. We fit our compiled data and label sets to one another using a multi-layer perceptron classifier (MLPClassifier) object from the FluCoMa Max library to generate our trained model [1]. During the process, we experimented with ideal training settings, verifying results with examples withheld from the training set. Our best-performing classification model featured two hidden layers of 12 and 8 neurons respectively, trained over 1000 cycles at a learning rate of 0.1 and 100 epochs, yielding a fit of 0.00137. For output, we transferred our trained classifier to a multi-layer perceptron regression (MLPRegressor) object built on the same architecture, adapting a workflow from Tremblay [18]. Instead of outputting only the predicted class, this approach gave us a five-point vector of the model’s predicted likelihoods for each trained category in response to new sensor data. We then divided this vector into five separate data streams, each representing the confidence rating for aspecificgestureclassonascalefrom0.to1.,andusedthesetocontrolvarious audio playback parameters. This hybrid workflow combining classification and regression tasks, one followed by the other in series, gave us the advantages of Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 942 Real-Time Gesture Classification for Performance a single classification model while providing the flexibility of multiple regression outputs, enabling smoother interpolation between classes along with affordances for harnessing the model’s predictions in continuous gesture-sound interaction. 50 ms Difference 16 point training vector Current Reading x y z x y z 1 2 Sampling Gyroscope Accelerometer EMG Features and Data Sampling for Classification Fig. 4. Data sampling workflow combining running averages for eight sensor values with differences in readings over 50 ms, resulting in a 16-point feature vector for training. 3.4 First Application and Testing In our first application of the trained model, we tested its performance in Max by mapping confidence ratings for each gesture class to the playback volume of audio clips associated with each gesture type (as described in section 3.2), so that a gesture’s associated sound would play more loudly when the model was more confident in that gesture. Audio clips were recorded via live improvisation into Ableton Live, edited, exported, and uploaded to Max. For smoother interpolation between audio clips in Max, we set individualized audio playback thresholds and volume scalings for each incoming confidence rating stream, and used “leaky bucket” first-in-first-out (FIFO) buffers to selectively slow the decrease of each rating within the patch. This allowed us to blend the influences of each rating on audio playback according to taste. We also performed additional smoothing on incoming ratings to reduce playback jumpiness, taking an average of the past five values. To test our system, author one performed gestural improvisations mixing examples of each gesture type with movements that fell between classes. We sent live IMU and EMG signals to the machine learning model using the same placements, format, and data processing used in training, outputting confidence ratings to control playback. We found that audio playback triggered by the model accurately reflected the trained gesture categories, responding reliably enough to allow for improvisation and play. While some false predictions occurred, practice with the system quickly reduced these and fine-tuning the patch also helped minimize them. The system also provided useful output for gestures that fell Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 943 E. O’Donnell and A. Tanaka between classes via overlapping or interpolated audio from its nearest guesses. This reduced empty auditory space and offered another area of exploration for the performer. In general, we found that the use of confidence ratings in place of direct classification output resulted in smoother transitions between sounds and allowed us to fine-tune the patch for more aesthetically pleasing and intuitive results. The overall mapping approach let us embrace the guesswork of the classification model in adapting audio output to variations in performance while keeping a firm basis in the chosen gesture-sound associations. Fig. 5. Screenshot of the Max patch used for playback testing in section 3.4 3.5 Creative Applications In lieu of a formal evaluation of this system, we incorporated the trained model into a series of composed Max performance patches, combining machine learning outputs with direct sensor mappings to work with a corpus of pre-composed rhythmand texture-based audio files. Building these performance patches allowed us to explore our sonic associations with the trained gestures in greater depth, along with alternative mappings for the model’s confidence ratings. Experiments included taking running averages of these ratings over longer time periods to influence the broader arrangement, using individual confidence rating streams in combination with direct IMU or EMG mappings to impact different parameters for the same track, and mapping gesture classification ratings to groupings of sound files based on subjective aesthetic judgments. In one experiment, the first author used confidence ratings for the “landing” category to add downbeats for phrase punctuation, and “stretching” ratings to Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 944 Real-Time Gesture Classification for Performance selectively slow playback rate, within an arrangement otherwise controlled by direct IMU and EMG data mappings. In another, author one mapped “agitation” confidence ratings to the volume of tracks he felt had a staccato rhythmic character, “circulation” ratings to tracks with a rumbling or rolling character, and the “idling” category to silence some tracks or increase the volume of a static drone to sit between phrases. We also explored triggering changes in the arrangement when one or several confidence ratings stayed above a given threshold for a long enough period. As part of a composed system created for several public performances in Fall 2025, the first author used the trained gesture recognition model to sculpt the sonic background for a percussive improvisation which used direct sensor mappings. During the composition process, author one grouped a corpus of precomposed sound files based on how well he felt they fit with each trained gesture type. He then adjusted the mix of these by ear, including assigning playback triggers for different files to different confidence thresholds, while mapping volume variations to different ranges in the confidence ratings. As a result, specific groups of files played in response to strong ratings for each gesture type, while other, less predictable combinations sounded for movements that fell between categories. A running average for each rating over a longer time interval (0.5 to two seconds) allowed for more gradual changes in this mix. In performance, as author one moved his arms to trigger separate interactions in the Max patch, the model continued to output predicted likelihoods for its trained gesture types to control this background mix, often without his conscious attention. While the latter implementation sometimes returned unexpected results, as performers we found that the system’s output often felt intuitive in relation to our executed movements. The use of our subjective or aesthetic associations in designing gesture-sound mappings resulted in interactions that reflected our embodied knowledge, while the ability to map the model’s regression outputs to both immediate and longer-term changes meant that a small number of gesture categories offered a surprising number of creative options. We ultimately found these shifting background mappings compelling enough to use on their own during some sections of the performance. As a whole, our creative applications of the system thus far demonstrated strong potential for this approach in crafting meaningful, complex, and subjective movement-sound mappings for composers and performers. The use of overlapping regression outputs from a single classification model offered a number of creative affordances, customizable to the performer’s embodied associations, with ample room for further exploration. 4Discussion Multimodal sensor data was crucial for distinguishing between gesture types without using time series classification. The combination of gyroscope, accelerometer and EMG readings gave us complex enough descriptors that a combination of running averages and differences over a short time interval were sufficient to differentiate one movement from another, without needing to describe gesture Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 945