Robo-Sax Ensemble System

Robo-sax ensemble system Gou Koutaki1[0000→0002→3414→1085] and Masatoshi Hamanaka2[0000→0001→5604→2582] 1Kumamoto University, Kumamoto, Japan [email protected] 2RIKEN, Tokyo, Japan Abstract. In this study, we develop a semi-automatic instrument-playing robot, Robo-Sax, which performs key fingerings automatically, while a human player performs the blowing motion. Additionally, we develop four types of Robo-Saxs (soprano, alto, tenor, and baritone) and propose a system that enables ensemble playing using a saxophone quartet. We seek to design the hardware and thoroughly develop the system software such that it can be utilized practically. Experiments using Robo-Sax are conducted with beginner, intermediate, and professional saxophonists to demonstrate this system’s potential. Keywords: saxophone ·robotics ·ensemble 1Introduction Music is a valuable component of our culture. In addition to listening to music, singing and playing musical instruments are used as hobbies; in education; and for fun, communication, and self-expression. However, the hurdles to playing these instruments are high, necessitating extensive periods of repetitive basic practice under an appropriate instructor’s guidance. Many people find this painful and often fail to learn. Consequently, this study aims to construct an inclusive music performance system that enables any person to independently play musical instruments with their hands and enjoy musical performances and ensembles without any di!- culty. Specifically, we develop a semi-automatic musical instrument-playing system that provides performance support using robotics and information technology, as well as enables expressions that are challenging for humans to perform (human augmentation). Although fully automated musical instrument-playing robots have existed for long [17,4,21], these devices are not fully automated; instead, half of the music is played by a human player, such that the player can experience realistic sound vibrations with their body without losing the original pleasure of playing [14]. This study develops a semi-automatic instrument-playing robot system for the saxophone. Moreover, the system realizes a saxophone ensemble by playing several of these robotic saxophones simultaneously. Use studies are conducted with beginner, intermediate, and professional saxophone players; finally, the system’s potential is discussed. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 736 2G.KoutakiandM.Hamanaka Fig. 1. Original alto saxophone and the Robo-Sax (alto saxophone) that we developed. Asteelsheetmetalembeddedwithservomotorsandelectroniccircuitsisfixedtothe saxophone and subsequently attached to the saxophone. There are 19 servo motors, which can cover all the registers employed in normal saxophone playing. Our contributions are as follows: 1) Development of a practical level semi-automatic saxophone robot: We meticulously studied the instrument robot’s mechanical mechanism and, after several prototypes, completed a saxophone robot with low latency and high safety. 2) Realization of ensembles with multiple Robo-Saxs: We created four di"erent robots (soprano, alto, tenor, and baritone). Thereby, the world’s first robot saxophone quartet was realized. 3) Applications: We provide some practical-use studies such as: -Example of Robo-Sax Performance by a Beginner. -Robo-Sax ensemble by members of the wind ensemble for one year and a concert by the Robo-Sax quartet. -Real-time melody generation and performance system by professional saxophonist. 2Relatedwork Numerous studies have been conducted on automatic musical instruments. This section introduces two types of robots—namely, fully automatic and semi-automatic instrument-playing robots. 2.1 Fully automatic musical instrument Atypicalfullyautomaticinstrument-playingmachineisthemechanicalmusic box [6], invented 200 years ago, followed by the hand-cranked organ[1], wherein musical sequences are provided using punch cards; in the early 20th century, Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 737 Robo-sax ensemble system 3 the Hupfeld Phonoliszt Violina [3], a machine for automatic piano and violin playing, was invented. These are types of mechanical music sequencers. After the 1960s, these mechanical sequencers were replaced by electronic sequencers, which became popular, along with synthesizers and sound modules. Research on robots that can play real musical instruments instead of electronic instruments was conducted in the 1980s. Thus far, robots that play the violin, piano, and guitar have been developed [11,5,13,2,18]. Automatic pianos have already been commercialized. Research on humanoid robots that can play trumpets and flutes has increased since 2000 [21,17,15,12]. Key operations are controlled by a finger-shaped robot, while breathing is controlled via artificial lips and lungs, allowing the robot to play actual wind instruments. Most fully automated performance robots are used for viewing. They are limited to passive musical experiences, such as a human listening to or watching arobotperform. 2.2 Semi-automatic musical instrument Machines and robots have been developed to assist in playing musical instruments semi-automatically rather than fully automatically. They are designed to educate people about playing musical instruments or for physically challenged people who experience di!culty playing musical instruments. Devices that assist in pressing a particular chord on a guitar, hold a violin bow in place, and attach to the singing mouth of a flute have long been employed as performance aids for beginners. Moreover, recorders that can be played with only one hand also exist. Recent research has examined musical instrument-playing support devices using robots and information technology [14,20,19]. Some devices use actuators to hold down the strings of a guitar or assist in the fingering of a flute [9]. In these semi-automated instrument-playing robots, because the main player is a human, the user can experience an active musical experience rather than merely a passive one, as in the fully automated type. 3Robo-saxsystem 3.1 Prior Knowledge: Saxophone [10] The saxophone on the left side of Figure 1 is a woodwind instrument invented by Adolph Sax in the 1840s. A wooden reed is attached to the mouthpiece; when the player blows into it, the reed vibrates to produce the fundamental tone, as depicted on the top-right side of Fig. 1. The tube length can be changed by closing the keys on the saxophone’s body, and the notes can also be changed. Approximately 20 keys can be operated by humans, and the pitch can be changed by approximately 32 semitones (two-and-a-half octaves). There are four types of saxophones—namely, soprano, alto, tenor and baritone. Each type has a di"erent range; however, the basic keys are identical. These are the transposed instruments. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 738 4G.KoutakiandM.Hamanaka 3.2 Division of roles between humans and robots We develop a semi-automatic musical instrument-playing robot called RoboSax. That is, the robot performs the key movements, while the human performs the blowing movements by breathing the sound in and out. Utilizing a semi-automatic robot, instead of a fully automatic robot, enables a performance experience. Conversely, a method exists wherein a human performs key fingerings and a robot blows the keys; however, we believe that this is not a desirable combination. This is because the act of blowing is directly associated with pronunciation and is more likely to provide a greater performance experience if performed by a human. This can be considered from another perspective. Specifically, the opening and closing of the keys, a 0-1 state change, is exactly the kind of movement that robots excel at; meanwhile, subtle breathing, a continuous state, is the kind of movement that humans excel at and is a major musical expression. 3.3 System requirements and design policy As fully automatic saxophone robots have already been developed [17], one might believe that a semi-automatic robot can simply draw functions from these robots. However, the answer is "No." If so, the semi-automatic saxophone robot we are aiming for must be handheld by a human. Specifically, the robot must be designed to satisfy the following constraints: (1) Small size and light weight. (2) Safety: no injuries when using it. (3) No interference with the player’s body. Fully automatic instrument-playing robots are not designed to be handheld by humans. Therefore, they do not satisfy these conditions. Additionally, the following conditions are desirable: (1) Low cost. (2) Not destroying the original instrument. (3) Easy maintenance 3.4 Hardware Fig 1 illustrates the appearance of the developed Robo-Sax. This is an alto saxophone robot. An iron sheet metal embedded with 19 servos and electronic circuits (microcontroller and servo driver) using screw holes in the saxophone’s thumb hook and thumb rest. The servo motors can open and close 19 keys, and the robot can play 32 semitones (two and a half octaves) of the range employed in normal saxophone playing 3. The power supply operates at 5V; hence, it can be operated outdoors using a smartphone’s mobile battery. The microcontroller has a type-C USB connection 3Except for a special technique called flageolet. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 739 Robo-sax ensemble system 5 and operates using MIDI signals. RoboSax is recognized as a MIDI external sound device and can be used in commercial DAW (Desktop Audio Workstation) applications. Fig. 2. Key-pressing method. (a) Solenoid is unsuitable for handheld robots because it is heavy and generates heat. (b) Servo motor is pulled by a wire, operates at a constant voltage of 5V, and can generate su!cient force at high speed. Initial trial: solenoid method Initially, we followed previous studies and tested the use of a solenoid (electromagnet) as the robot actuator to open and close the key, as illustrated in Figure 2 (a). However, this method exhibited the following problems: (1) The solenoid including the iron core is heavy. (2) A high voltage of approximately 24V is required. (3) To keep pressing the key, continuously applying voltage to the solenoid is necessary. This causes solenoids to heat up, occasionally exceeding 100↑C. The robot weighed 4kg and required an aluminum frame for support. Therefore, the solenoid method is unsuitable for our handheld semi-automatic instrument. Proposed method: Servo and wire method We developed a method to open and close the key using a small DC servomotor using a low-voltage 5V and a wire. As illustrated in Fig. 2 (b), the key is connected to a wire attached to the end of a servo horn, and the wire is pulled by the servo’s rotation. We discovered that nylon-coated wire used in handicrafts can be utilized as wires. This wire is strong and flexible and can be used with a special fixing part called “crimp beads" and a special tool. Various types of Robo-Saxs Various types of saxophones exist—namely, sopranos, altos, tenors, and baritones. We also developed robots for these saxophones. We employed fundamentally the same design principles as those used to Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 740 6G.KoutakiandM.Hamanaka Fig. 3. Various types of Robo-Saxs. From left to right are the soprano, tenor, and baritone saxophone robots that we developed. create the alto saxophone robot described in the previous section. The soprano saxophone is smaller than the alto saxophone, whereas the tenor and baritone saxophones are larger. The baritone saxophone is nearly one meter in length. As the prototypes were being made, the tenor and baritone saxophones did not produce a satisfactory sound with the servo-and-wire method. This was because the saxophones had become larger, and the wire did not have su!cient force to hold the keys down. Moreover, another problem was identified. As depicted in Figure 1, the alto saxophone robot is attached to the back of the saxophone. When the robot was attached to the back of the tenor or baritone saxophones, it collided with the performer’s body during the performance, resulting in interference. Therefore, for the tenor and baritone saxophones, the keys were pressed with arodusingrackandpiniongears,asdepictedinFigure4.Furthermore,arobot was placed between the body and bell, and the back of the saxophone was the same as that of the original saxophone, making it possible to press the keys with su!cient force and solving the problem of body interference with the player. 3.5 Software The Robo-Sax developed can be used with a MIDI device, and the keys of the Robo-Sax can be moved by playing the saxophone track of a MIDI song using DAW software. However, because Robo-Sax is a semi-automatic instrument robot, a human must perform the role of blowing into the mouthpiece to produce sound. Additionally, the player must adjust the pressure of biting the reed with the mouth and its muscles according to the saxophone’s range. These are called “embouchure.” Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 741 Robo-sax ensemble system 7 Fig. 4. The tenor and baritone saxophones use a rack and pinion system to hold the keys, with the robot positioned between the body and the bell. Fig. 5. GUI software for Robo-Saxs ensembles. Robo-Sax functions when the notes flowing from above reach the red line in a music game format. Thus, considering a means to synchronize the timing between the human and the machine is necessary, as is telling the player which notes to play. Furthermore, while multiple Robo-Sax can be employed simultaneously to form an ensemble, a system for time synchronization of multiple performers is required. This is challenging because general DAWs are not designed for semi-automatic instrumentation. Therefore, we developed a unique GUI software suitable for Robo-Saxs ensembles. Their appearances are depicted in Figure 5. In the form of a so-called music game, the system visualizes the scale notes of each part of multiple RoboSaxs and plays background or clicking sounds, enabling players to synchronize their timing visually and aurally. 4 Experimental results 4.1 Use Studies Examples of the use of the RoboSax system are as follows: Example of Robo-Sax performance by a beginner Two students who had never played the saxophone before played the robot alto saxophone. They had never played the saxophone before and could not read sheet music. Although the saxophone is a relatively easy instrument for beginners to play reed sounds, we Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 742 8G.KoutakiandM.Hamanaka Fig. 6. Robo-Sax performance by a beginner. Despite having no saxophone experience at all, the two were able to play the tune on the saxophone. gave them a lecture on playing the reed sound with a mouthpiece for the first 30 minutes, after which they played the Robo-Sax. The music score to be played is illustrated on the left of Figure 6, while the performance is depicted on the right. Despite having no prior saxophone experience, the two could play the tune on the saxophone. Usually, beginners in a performance are occupied with merely producing sound, and if they must move their fingers simultaneously, it typically takes some time before they can play a tune. However, this experiment confirmed that even beginners can play a tune in a short time using the Robo-Sax. In an interview after the performance, the students commented that they enjoyed producing saxophone sounds and playing songs. The Robo-Sax enables students to practice key fingerings and blowing independently and keeps them motivated because they can play their favorite songs. Fig. 7. Robo-sax ensemble by members of the wind orchestra club. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 743 Robo-sax ensemble system 9 Robo-sax ensemble by members of the wind orchestra club With the assistance of seven members of the saxophone section of the Kumamoto university’s wind orchestra ensemble club, we performed experiments with performing ensembles with the Robo-Sax over one year from March 2024 to March 2025. Practice sessions were held once a month for two hours. The composition was an SATB (soprano, alto, tenor, and baritone) saxophone quartet. Depending on the song to be played, the band was made into a saxophone quartet of AATB formation, with two alto saxophone robots that switched from soprano to alto. In November 2024, a mini-concert was held for the public to check the results of the practice and motivate the club members. The long-term RoboSax experiment allowed us to identify problems and refine the system. Figure 7 (a) presents club members using RoboSaxophone for the first. Initially, the club members were puzzled; however, eventually, they could perform as an ensemble without major problems. a) Feedback from players In an interview after the practice session, the following comments were obtained: -It is extremely challenging to find the right timing to blow in the air. Tangling and staccato playing are impossible. -Di!culty in GUI in the form of a music game. -Robo-Sax is heavy and cannot be supported for a long time. b) Improvements from feedback Adjustment of delay: Athoroughrobot-humantimingadjustmentwasperformed. This was because the timing for blowing in the breath did not match the movement of the keys. That is, if the breath is taken in while the keys are in the process of closing, the saxophone will not produce a sound or turn over. The mechanical delay of the robot, MIDI communication, and display delay are not negligible. If the amount of delay is constant, it can be canceled by sending the control signal earlier by the amount of delay. The performance quality was improved by adjusting the delay. Introducing paper printed music sheets: This is because wind club members are usually accustomed to playing music with stave notations. They required information on musical notation to change the embouchure according to the pitch of the notes. Therefore, we prepared paper-printed music sheets and placed them on music stands. Introducing the playing stand: As the robo saxophone is heavy, playing it for prolonged periods with the saxophone strapped around the neck or shoulder, as usual, caused significant fatigue. Therefore, for the soprano and alto saxophones, placing the RoboSax on a tuba stand reduced fatigue. Tenor saxophones were not a major concern because they can be strapped to the shoulder. The baritone saxophone was placed on a saxophone stand. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 744