Real-time audio interaction in serious games for music learning

REAL-TIME AUDIO INTERACTION IN SERIOUS GAMES FOR MUSIC LEARNING Isabel Barbancho Lorenzo J. Tard´ on Ana M. Barbancho Universidad de M´ alaga, ATIC Research Group, Andaluc´ ıa Tech, E.T.S.I. Telecomunicaci´ on, Dpto. Ingenier´ ıa de Comunicaciones, Campus Universitario de Teatinos s/n, 29071, M´ alaga, Spain [email protected], [email protected], [email protected] ABSTRACT In this LBD, we present several Apps for playing while learning music or for learning music while playing. The core of all the games is based on the good performance of the real-time audio interaction algorithms developed by the ATIC group at Universidad de M´ alaga (SPAIN). 1. INTRODUCTION The Serious Game concept is used to describe games designed to serve an additional purpose to that of pure entertainment: attaining specific learning outcomes, psychological or physical stimulation or skill by education, advertising, training or simulation. Even before the wide success of computer games, the term serious game had been introduced in [2] and nowadays is well known. Consequently, gaming technologies are becoming more and more important because they provide serious games with state-of-theart technologies to better serve the serious game purposes: augmented reality, 3D modeling, virtual reality, etc. and, of course, music information retrieval. In this LBD, we present how the ATIC group [1] is using some of the audio transcription algorithms developed trough the years [4], [3], [5], to serve in serious games for music learning. In this research group, there are not only people with technical and scientific knowledge, but also people with professional music knowledge. This fact has made possible the implementation of serious games for music learning that are beginning to be used in some Music Schools. 2. APPS FOR MUSIC LEARNING WHILE PLAYING In this section, we present several Apps in which several of our music information retrieval algorithms are used. c Isabel Barbancho, Lorenzo J. Tard´ on, Ana M. Barbancho. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: Isabel Barbancho, Lorenzo J. Tard´ on, Ana M. Barbancho. “Real-Time Audio Interaction in Serious Games for Music Learning”, Extended abstracts for the Late-Breaking Demo Session of the 17th International Society for Music Information Retrieval Conference, 2016. 2.1 Sing’n ’Colour In this App, available at the Apple’s App Store, the users learn music while coloring different scenes and objects by singing. Each color is associated to a musical note. The aim is to sing the notes correctly to color the pictures in the way the player is indicated at the beginning of each exercise. The number of different colors to sing depends on the level of difficulty. This game is well suited for small children. They only need to tune properly a note, without constraints regarding when and for how long the note should be sung. Figure 1. Sing’n’Colour App screenshots. Available at App Store https://itunes.apple.com/us/app/singncolourlearning-music/id668360474?mt=8 2.2 Sing & Draw Sing & Draw is an interactive music learning game designed to guide you to learn how to sing musical notes in tune while creating your own drawing. The player has to sing in tune, in any octave, one of the seven notes: C/Do D/Re E/Mi F/Fa G/Sol A/La B/Si to select the color while drawing by swiping the finger on the screen.The color depends on the pitch, i.e., on the sung notes, regardless the words of the sung song. Figure 2. Sing & Draw screenshots. Available soon at App Store. 2.3 Sing & Eat (CantaCome) CantaCome (Sing & Eat) is an interactive music game to help you learn to tune musical notes as you play. The cow moves according to the musical note sung: C, D, E, F, G, A, B. There are no note figures and the audio engine recognize this seven notes. Figure 3. Sing & Eat App screenshots. Available at App Store https://itunes.apple.com/us/app/canta-comepro/id927300006?mt=8 2.4 Note Hunters In this case, the game can detect a wider range of notes thatn in previous Apps, from MIDI=55 (G2) to MIDI=77 (F4). In the interface, a pentagram is included with the corresponding note figures. There are no measures separation though. The game is suited for kids and women voices or for men who are able to sing in those frequencies. Stella, the character, flies up or down depending on the note that the player signs. 2.5 Magical Music This last serious game is the most evolved one regarding musical notation and performance. This game offers a Figure 4. Note Hunters screenshots. Available soon at App Store and Google play. complete correct musical notation including note figures and measure division. Even more, the game offer the possibility of choosing the type of voice (kid, women and men) so that the MIDI notes range is adapted to the singer (player). Figure 5. Magical Music screenshots. Available soon at App Store and Google play. 3. CONCLUSIONS Several Apps for music learning have been presented. Depending on the age and the users’ skill, a different one can be used for learning or improving several music skills while playing. The success of all of them relies on a nice graphical interface, a good adaptation of pedagogical method for music learning and on the performance of the real-time audio interaction algorithms developed by the ATIC Research Group. 4. ACKNOWLEDGEMENTS This work has been funded by the Ministerio de Econom´ ıa y Competitividad of the Spanish Government under Project No. TIN2013-47276-C6-2-R and by the Junta de Andaluc´ ıa under Project No. P11-TIC-7154. The work has been done at Universidad de M´ alaga. Campus de Excelencia Internacional Andaluc´ ıa Tech. 5. REFERENCES [1] ATIC Research Group. http://www.atic.uma.es. [2] Clarc C. Abt. Serious games. The Viking Press, 1970. [3] E. Molina, A. M. Barbancho, L. J. Tard´ on, and I. Barbancho. Evaluation framework for automatic singing transcription. In International Society for Music Information Retrieval (ISMIR), pages 27–31, 2014. [4] E. Molina, L. J. Tard´ on, A. M. Barbancho, and I. Barbancho. Sipth: Singing transcription based on hysteresis defined on the pitch-time curve. IEEE Transactions on Audio, Speech, and Language Processing, 23:252– 263, 2015. [5] C. Roig, L. J. Tard´ on, I. Barbancho, and A. M. Barbancho. Automatic melody composition based on a probabilistic model of music style and harmonic rules. Knowledge Based Systems, 71:419–434, 2014.