Legato: una herramienta de música adaptativa para videojuegos inspirada en técnicas cinematográficas

Legato: una herramienta de música adaptativa para videojuegos inspirada en técnicas cinematográficas Legato: an adaptative music toolset for video games based on cinematic techniques Trabajo de Fin de Grado Curso 2023–2024 Autores Emile de Kadt Pablo Etayo Rodríguez Directores Jaime Sánchez Hernández Miguel Gómez-Zamalloa Gil Grado en Desarrollo de Videojuegos Facultad de Informática Universidad Complutense de Madrid Legato: una herramienta de música adaptativa para videojuegos inspirada en técnicas cinematográficas Legato: an adaptative music toolset for video games based on cinematic techniques Trabajo de Fin de Grado en Desarrollo de Videojuegos Autores Emile de Kadt Pablo Etayo Rodríguez Directores Jaime Sánchez Hernández Miguel Gómez-Zamalloa Gil Convocatoria: Junio 2024 Grado en Desarrollo de Videojuegos Facultad de Informática Universidad Complutense de Madrid 27 de mayo de 2024 Dedicatoria A Jaime, por disfrutar tanto de lo que enseña y a Miguel, por ponerle los pies en el suelo cuando toca. v Agradecimientos A todo ASCII, por mantener mis ganas de vivir estos cinco años. -Emile A mi familia, por animarme a esforzarme durante estos años de carrera. -Pablo Bueno, sí, a mi familia también, claro está. -Emile vii Abstract Legato: an adaptative music toolset for video games based on cinematic techniques This project provides video game developers with a set of tools to create adaptive soundtracks for their games, which react to game events, using techniques inspired by cinema scores. The techniques that inspired this project (and the ones it therefore facilitates) revolve mainly around the construction of a “musical identity” for different aspects of a scene or narrative. A certain in-game location may have an associated instrument within the soundtrack; a terrifying villain might cue an ominous leitmotif; the game’s music may slow down drastically during key player decisions. With the tools provided by this project, elements within the game can easily be assigned a musical identity, affecting the soundtrack in a dynamic manner. Many sections of this document assume a basic familiarity with the Unity game engine on behalf of the reader. Keywords Adaptive music, video game engines, Unity, Reaper, musical composition, video game music, OSC. ix Cap´ ıtulo 1 Introduction “If you talk to any director, they’ll say music is fifty percent of the movie.” — Hans Zimmer 1.1. Motivation Even in their very first projections, the Lumière brothers relied on a pianist to accompany their short films (Tudor, 2017). Predating even the introduction of dialogue to the medium, the soundtrack has always been an indispensable part of cinema. It is hard to picture the great protagonists of cinema without evoking the music that heralded their presence. Don Corleone’s tragic violin segment; the thunderous Imperial March of the Galactic Empire; the flute, ocarina and choir that define the eponymous “The Good, the Bad and the Ugly” despite playing the same tune (Classic FM, 2014); the full cast and story of the Spider-verse, made that much more vibrant by the array of techniques, instruments and waterfowl that Daniel Pemberton spun into these films (Sony Soundtracks, 2023). The identity of these characters is just as audible as it is visual. Many of these famous sounds are leitmotifs, melodic figures or phrases associated with the repeated appearance of a character, idea or situation. Others, like “The Good, the Bad and the Ugly”, use instruments to identify their characters. This concept of “musical identity” fascinated us. As game development students, we wanted a way to employ these techniques in our own games. The soundtrack of a videogame has an allure comparable to that of a film; however, save for certain exceptions like rhythm games, the sequence of events that occur in a game is not deterministic. The decisions of one or more players, as well as other random events, make it impossible to compose songs that perfectly resonate with each moment of the action. Adaptive music as an art-form rose as a way to patch this issue: a series of editing and sequencing techniques that alter a game’s music in real time to better follow the events on screen. The music may be altered by changing the instrumentation, adjusting the tempo (the speed at which the music is played) or introducing new 1 2Capítulo 1. Introduction melodies into the current song. We had a very clear mental image (or rather, mental sound) of what we wanted: a videogame score that uses musical language to describe the action on screen at any given time, by combining signature character leitmotifs and instruments in harmony with each other. As game developers, we looked for middleware (that is, software that works underneath the game’s code but above the operating system) that would allow us to create music for our games straight from the game engine using these techniques. Unfortunately, we could not find anything that quite matched our specifications. Composers and game developers use such tools as DAWs and trackers (two different types of applications used for producing music) to create and integrate soundtracks into games. However, these applications still require human intervention and can’t necessarily be controlled from the game engine. This is important, since a game’s logic is programmed within the engine; it would be unwieldy to have to edit the source files of the soundtrack every time a change in gameplay affects how and when the music will be played. Without a specialised tool, the functionality we want would become very messy to implement. Every possible combination of leitmotif and instrument would need to be rendered individually in every tempo. Even in specialised environment, such as a DAW, this task would become tedious without automation. Additionally, synchronising a soundtrack made up of so many scattered pieces would be it’s own struggle without a centralised handler for the music. These considerations helped us narrow down the requirements of the work needed to create such a tool ourselves: an adaptive music system that borrows musical techniques from cinema, allowing for complicated, high-cost instruments and effects without demanding large amounts of processing power from the player’s computer. 1.2. Key definitions Horizontal re-sequencing: set of adaptive music techniques in which different audio clips are played one after the other, changing the order or choice of clips in response to events within the game. Vertical re-orchestration: set of adaptive music techniques in which different parts of the soundtrack that are played simultaneously are altered in response to in-game actions, resulting in a different final mix. MIDI (Musical Instrument Digital Interface): digital encoding format for music used for communication between applications or devices. MIDI items on their own store no information about the music’s sound; only the about the note that is played, as well as additional data like the timing and way in which the note is played. DAW (Digital Audio Workspace): device or software used for recording, editing and producing sound audio. Among other things, a DAW allows for 1.3. Goals 3 the rendering of sound files from an encoding format (such as MIDI) and list of effects that combine to determine what sounds are produced by each note. Virtual instrument: software for generating sound from a symbolic representation (within the scope of this project, generally MIDI). These instruments serve as plug-ins within a DAW. OSC (Open Sound Control): network protocol for multimedia communication between sound devices. Crossfade: to change from one sound to another via a gradual fading out of one clip as another is faded in. Quantisation: within the context of adaptive music, the process of correcting and synchronising different sources of music so that they sound perfectly aligned with their tempo. BPM (Beats Per Minute): unit of measurement of the music’s tempo. Positional audio: in videogames, any sound that simulates coming from a specific point within the game space. 1.3. Goals The aim of this project is to create an extension for Unity that would include tools for the inclusion of adaptive music in videogames using the above mentioned techniques. Unity is a game engine with a very wide user base. It was mainly chosen due to our familiarity with it, though it has other advantages, as explored in chapter 2. Since the tool is built within and for use with Unity, it is also targeted at Unity developers. The tool should simplify the process of building a musical identity for different characters, themes and locations in the game’s narrative, by altering the ambient soundtrack whenever those narrative elements are in a scene. While the actual composition of the music should still occur within a DAW, a developer should be able to make any changes to the logic of how the soundtrack is used without leaving the game engine. This toolset should: Allow for the modular inclusion of leitmotifs into a game’s soundtrack. Allow parts of the soundtrack to have their instrumentation changed in runtime. Allow for the tempo of the soundtrack to be changed in runtime. Place a low demand of resources on the game engine. Not require composers to understand any programming. 4Capítulo 1. Introduction Not require programmers to understand music generation software. Place as few restrictions as possible on the music and effects that the composer can create for the game. Be easy to use. 1.4. Work plan In order to fulfil these objectives, the work was split into the following parts, some of which were developed in parallel: Research of existing technologies. Rather than write all of the code from scratch, different applications were studied to be used as a foundation upon which to build the tool. This would also make the tool more accessible, by linking it with existing technologies that potential users would already be familiar with. Automating the rendering of sound files. Since the tool will use pre-rendered sound assets, these will need to be created automatically based on user-established parameters. Creation of the user interface for the tool’s configuration. The developer should have access to a control panel within the game engine itself. Creation of game assets that could represent the dynamic elements of the soundtrack. To make the tool more visual, the adaptive leitmotifs, instruments and tempos will be represented as assets within the game engine. These assets will need to be regenerated every time new audio samples are rendered. Programming the dynamic playback logic. The adaptive playback system is the core of the tool. Other than ordering the entrance and exit of the dynamic elements, the developer should not have to manually control the synchronisation, quantisation or playback of the music. Cap´ ıtulo 2 State of Play Legato is a toolset for creating adaptive music for videogames. As such, it would be pointless to study other similar tools (its “competition”, as it were) without first examining a few illustrative examples of adaptive music in popular videogames. 2.1. Adaptive music in videogames While the following examples do not communicate the full capabilities and limitations of the tools used to create them, they do paint a picture of the general requirements that any similar tool must fulfil. This of course cannot be considered an exhaustive analysis of all the techniques used in the medium, but it highlights a fairly representative selection. Doom Eternal (id Software, 2020) is a high-action, first-person shooter game with a heavy metal soundtrack. Its score employs a variety of techniques to capture the game’s energy of frantic, non-stop action: •A vertical re-orchestration system that uses different tracks to score different levels of combat intensity. •A horizontal re-sequencing system that transitions between different songs when the player moves between scenes. These transitions are occluded by additional diegetic and non-diegetic sound effects that play over them. •When the player’s health is low, a low-pass filter is applied to all of the game’s music. This is a common effect across the action game genre. (Porkbrain, 2020) Mario Kart 8 (Nintendo, 2014) is a multiplayer racing game with a unique sountrack for every racetrack. The track Mount Wario makes notable use of horizontal techniques; the level uses 4 different songs that play during their respective parts of the track. Each time a player reaches a new zone, the currently playing song continues until the end of its next measure. Then, it transitions directly into a percussion-only bridge that leads into the next song (8-bit Music Theory, 2019). 5 6Capítulo 2. State of Play Dead Space (Motive Studio, 2023) is a survival/horror game with third-person shooter mechanics. The soundtrack makes use of “aleatoric” techniques in the performance of the music. This on its own is not adaptive music, since the randomness is baked into the recording, not generated during gameplay (Pepiatt, 2023). Luftrausers (Vlambeer, 2014) is a 2D arcade-style game in which the player controls a customisable fighter plane, dodging attacks and shooting down other targets. The plane has three parts that can be customised. Each of these parts has a corresponding layer of the soundtrack, and each selectable option has an associated track. This way, each of the 125 possible combinations of parts creates a unique plane with a unique soundtrack (Ligman, 2014). The Legend of Zelda: Skyward Sword (Nintendo, 2011) is a medieval fantasy adventure game in which the player must explore the map and its various dungeons. Some of these dungeons employ a vertical re-orchestration technique: they have a single ambient track throughout the whole dungeon that has specific motifs layered on top based on the current room the player is is (Game Maker’s Toolkit, 2014). Wii Play (Nintendo, 2006) is a game with 9 mini-games, each one focused on highlighting certain capabilities of the Wii Remote. The last of these minigames, Tanks!, allows one or two players to partake in toy tank battles. The soundtrack to this mini-game employs a combination of vertical techniques: a single, short loop is played over several layers of music. Each enemy within the game determines what track plays on layer. The music is affected not only by which types of enemies are present, but also by how many there are and whether that is the strongest enemy remaining (Scruffy, 2022; isavenewspapers8890 y BigKitty1011, 2022). 2.2. Existing technology It is often difficult to identify which tools were used to program the adaptive music of a game; moreso, transitions between states of the music are frequently programmed directly from the game’s code, without the use of specialised tools (middleware). 2.2.1. Sequencers Sequencers were widely used in the 80’s, as a very efficient way of compiling a game’s soundtrack. In a time were every kilobyte was precious, storing an entire sound file in a game cartridge was too costly. Instead, music was generated with a soundchip and a list of notes, similar to how a music box functions. Since the music 2.2. Existing technology 7 was interpreted in real-time, this made it very easy and efficient to change the speed or instrumentation of the music. However, sequencers don’t scale well: the old games that used them (as well as newer games looking to emulate the retro"feel) used very simple synthesizers to play the game’s music. By contrast, modern virtual instruments can use large amounts of memory. If Legato is to be compatible with any sound the user wishes to use, it has to use pre-rendered audio. With this in mind, it is worth exposing the capabilities of three tools with particular relevance to this project: 2.2.2. Reaper Reaper is a DAW that allows a great amount of freedom for sequencing music using virtual instruments and other effects. These instruments can include any VSTi plugin, which is the commercial standard for virtual instruments. The program can be managed with a wide array of actions, unitary instructions each with a very specific functionality. These actions do not receive arguments, though in some cases they deploy pop-up widows for the user to allow the user to manually configure certain parameters. Some of these actions are tied to keyboard shortcuts for their ordinary use. For example, the action “File: Save project” is associated to the keyboard shortcut “Ctrl + S” (CockOS Wiki, 2013). In addition to its actions, Reaper has its own scripting language called ReaScript. ReaScript can be written in EEL2, Lua or Python. We opted to use Lua, since it is the most easily accessible from within Reaper itself and due to our familiarity with the language. ReaScript is very poorly documented; the vast majority of its functions have only a handful of words to describe them, and very few of them offer any explanation about their parameters. A video by Julio Carreño was the most useful resource for learning how to use ReaScript in any capacity (Carreño, 2023). Reaper has the ability to receive instructions via OSC (Open Sound Control). This is designed primarily to facilitate communication with electronic or digital instruments. However, for this project, it also proved a useful means of cross-application communication. One of the project’s goals is to interfere as little as possible with the creative opportunities available to the users of the toolset. 2.2.3. FMOD Studio As a sound effects engine, FMOD not only has a very accessible payment model, but it is one of the most powerful and widely used tools on the market. These strengths are not limited to adaptive music, as FMOD is useful for most all audiorelated tasks in game development. In the early stages of the project, FMOD was considered as a great potential foundation upon which to build the tool. However, it eventually proved unreasonable, as the additional layers of automation required to establish communication with FMOD studio, as well as create FMOD projects out of the samples rendered in 8Capítulo 2. State of Play Reaper, would have required more work than simply recreating the main functionality that FMOD would have provided for this particular tool (namely quantisation). At the same time, using FMOD would have placed additional restrictions upon the end user, forcing them to use FMOD for the whole project (since FMOD’s Unity plugin is incompatible with Unity’s native sound engine). This on its own would not have been a major drawback though, due to FMOD’s existing popularity. 2.2.4. Unity Unity is a component-based game engine with a visual editor for building game scenes. Is is used extensively across the industry for its familiarity, popularity, clear documentations and wide community support. Unity’s audio engine, like Unity itself, is not the most powerful option, but it is versatile and easy to use. The main parts of Unity’s audio engine (as concern this project) are the following: The audio clock. This internal clock tracks time much more precisely than the rest of the engine, since it is based on the number of audio samples that are processed. The AudioSource component. This component plays a single audio clip. It has configurable volume and positionality values as well as public functions to pause, play, stop and change the clip. One noteworthy function is the PlayScheduled method, which allows the user to choose an exact future moment in time (based on the audio clock) for the clip to begin playing. However, this function has a weakness: since the component can only store one clip at a time, the component cannot simultaneously play one clip while another is scheduled to begin playing. The AudioListener component. This component determines the relative position from which all positional sounds in the scene are heard. In order to function correctly, there can be no more than one in the same scene. Audio Mixers. Unity’s mixers are a type of asset that allows for different audio sources to be mixed and have effects applied to them. An audio mixer can receive input from multiple AudioSource components as well as other AudioMixers. A wide variety of tools can be found in the Unity Asset Store when exploring the audio section. There are tools for analysing songs to extract information about them (HelloMeow, 2024), tools for creating more realistic 3D sound (Techmo Voice Technologies, 2019), for rhythm visualization (Legend, 2020)... but there are no existing tools related to the incorporation of leitmotifs. Cap´ ıtulo 3 The Toolset Legato is a package that can be downloaded from github (de Kadt y Etayo, 2024). It allows for the integration of an adaptive music system into a project’s scenes. With the addition of a single Unity component, a game object can be granted one or more musical elements, which can be cued into the soundtrack with a single public method call. All of the scripts provided in the package (with the exception of the ones that are used exclusively in the demo scenes) are contained within the namespace Legato. The tool consists of two main parts: one to facilitate the rendering of the required samples in Reaper and another to allow the configuration and execution of the tool in Unity. Chapters 3 and 4 assume a background in Unity for a full understanding of their content. 3.1. Automated rendering in Reaper In order to function, the tool requires audio resources with a specific format and name structure. Included as part of the package is a custom Reaper action that receives a set of specifications from the Unity project and renders all the necessary audio files. For this action to work properly, the Reaper project must be set up in the following format: The first tracks to appear must correspond one-to-one with the different instruments that will be used in the game’s adaptive soundtrack. The instrument tracks must contain no MIDI or sound elements, only virtual instruments and other effects. There is no additional limitation to the sounds that can be used as instruments: any combination of effects that Reaper is able to process is valid. The remaining tracks must correspond one-to-one with the leitmotifs that will be used in the soundtrack. These tracks can contain effects, but these will not appear anywhere in the final renders. Both the instrument tracks and the motif tracks must appear in the same order 9 16 Capítulo 3. The Toolset when there are no additional motifs queued from game events. Here it is important to define the distinction between a fragment and a motif within the context of the channel: a motif is an object of the Fragment class that is queued into the emitter by an external object. Motifs are stored in the channel in an internal dynamic priority queue and always take priority over the rest of the fragments. All motifs are fragments, not all fragments are motifs. Algorithm The selection algorithm that is used to arrange the fragments from the previous array. There are three options to choose from a drop-down: •CircleQueue: the fragments are played in the order they appear in the array, looping back to the start after the end. •Random: each successive fragment is chosen at random from out of the whole array. •Shuffle: the fragments are played in a random order; once all fragments have been played, they are compiled into a new random order and played again. Volume Controls the channel’s volume. Instrument Determines the instrument that this channel will be played in at the start. Tempo Determines the tempo that this channel will be played in at the start. BeatsPerMeasure Determines the number of beats per measure. This doesn’t alter the sound produced by the individual fragments, it only affects the quantisation used for synchronising channels and changing between fragments at the right time. PlayOnStart Boolean value that determines whether the channel will begin playing at the start of the scene, or remain inactive until it is given the command to start. 3.3.3. Public interface of the Emitter Aside from the visual interface in the Unity editor, the emitter has public methods that allow for the following functionality: Change the volume of the emitter, or of a specific channel in the active sound scene. Change the active sound scene to another one in the SoundScenes array. Change the spatialisation values of sound scenes. 3.3. Emitter 17 Activate a motif in a specific channel. Cancel a motif that is queued but not playing yet, or interrupt one that is already playing. Pause, resume or stop one or all channels in the current sound scene. Stopping a channel means that when it is resumed, it will pick up at the start of the next measure, playing its next fragment. A paused channel will resume exactly where it left off, and may fall out of sync with other channels when resumed. Change the instrument of a channel in the active sound scene. Change the tempo of a channel in the active sound scene. Failed calls (such as attempting to change a channel to an instrument that the current sample is not rendered in) will produce clear error codes in the Unity log, specifying the rendered asset that was not found. 3.3.3.1. Dynamic priority of motifs The goal of this process is to help make motifs play as close as possible to when they are needed, in situations where the game may be triggering a large number of motif events in a short span of time. Mainly, it aids in two specific cases: When it is important for a motif to sound close to the moment it was triggered, because it loses meaning if it sounds too late. This would call for a gradual increase in priority. If a motif is only worth playing close to the moment it was triggered, and becomes less relevant over time. This would call for a gradual decrease in priority. When a motif is queued by a game event, it can be assigned a priority value. At the start of the next measure or at the end of the current fragment (determined by the user) the motif with the highest priority value in the queue (if any) will play. In addition to its initial priority value the motif can be given an increase or decrease in priority over time. Both of these values (the initial priority and the increase/decrease priority) must be given at the moment the motif is queued. This value is interpreted in units per second. This means that if a motif is given an initial priority value of 5 and a priority over time value of -1 will have a priority of 2 at the point in time 3 seconds after it was queued. A motif that ever reaches a priority equal or less than 0 will be discarded from the queue. 18 Capítulo 3. The Toolset 3.3.3.2. Instrument crossfading Any change in a channel’s instrument can be configured to occur suddenly or gradually, via a crossfade. In neither case will the change happen the instant the event is triggered, as it is necessary to first load the clip using the new instrument (this process typically takes a fraction of a second). An instrument change can be programmed to begin as soon as possible or at the start of the next fragment. In the latter case, the next fragment will begin playing with the new instrument, without crossfading or any additional loading time. If additional instrument changes are triggered before a previous one finishes, the last of the triggered changes will be stored and take effect when the current change is finished. Figure 3.8 illustrates how the program handles multiple successive instrument changes on the same channel. Each instrument is represented by a colour, with the vertical dashed lines representing the moment in time that an instrument change is requested. The curved lines represent the crossfade itself, as one instrument fades out and another fades in. In figure 3.8, instrument A (blue) is playing in the channel at the start. In order: 1. The channel receives a request to crossfade to instrument B (orange). 2. After a brief loading time, the crossfade begins. 3. During the crossfade, the channel receives another request to change instrument to instrument C (red). Because the channel is already in the middle of a crossfade, the change to C is postponed until the end of the current crossfade. 4. After this, but still before the end of the first crossfade, the channel receives yet another request to change instrument, this time to instrument D (purple). Just like with C, the change to D is postponed until the end of the current crossfade. However, this change overwrites the change to instrument C. 5. Once the A ->B crossfade is finished, the channel detects that there is still another crossfade queued (to instrument D). So, once again, it prepares to crossfade by loading the appropriate clip for D. 6. During this time, the channel plays using only B, until the B ->D crossfade begins. 3.3.3.3. Tempo changes Tempo changes are not as configurable as instrument changes. When the program triggers a tempo change in the current scene, or in a specific channel, this will take place at the start of the next measure. If multiple changes in tempo are triggered within the same measure, the last one triggered will be the one that takes place, exactly as if all the previous ones had not been triggered at all. 3.4. Assets 19 Fig. 3.8: Graphical representation of multiple consecutive instrument changes. 3.4. Assets The dynamic elements of the soundtrack (fragments, instruments and tempos) are all represented within the Unity project as assets. These classes cannot be created manually, only using the Render and Regenerate Assets buttons in the configuration menu. These assets make it easy to assign objects from the editor. Clicking on the Instrument field of a channel opens a visual menu (figure 3.9) that allows the user to select one of the rendered instruments. Another option is to drag an instrument from the Assets/Resources/Legato/Instruments to the desired field. The same process applies to fragments and tempos. Fig. 3.9: Instrument selection menu. 20 Capítulo 3. The Toolset When referencing one of these objects from code (for example, to request an instrument change), the user has the option to pass the asset directly, or to pass its name as a string (or int, in the case of tempos). The latter method is less efficient though. 3.5. Auxiliary components To complement the emitter, the tool also includes five components that simplify the process of activating events. These are Legato_InstrumentEvent,Legato_MotifEvent,Legato_TempoEvent,Legato_StopEvent and Legato_ResumeEvent. These components have a single public virtual method Trigger, which serves to activate the associated instrument, motif or tempo or to stop or resume specific channels, respectively. Fig. 3.10: Editor view of the five classes that inherit from Legato_Event. 3.6. Demo scenes 21 These five classes inherit from the abstract class Legato_Event, allowing them to be used polymorphically. Each component has additional parameters (figure 3.10) that are visible from the editor. Legato_ResumeEvent has a parameter Only Resume Previously Playing which, if marked along with Apply To All, makes the event only resume tracks that were playing before a call was made to stop all channels. 3.6. Demo scenes The package includes two demo scenes, found in Assets/Legato/Examples/Scenes. 3.6.1. Legato_Pad This scene contains an emitter and a set of buttons that modify the music played by the emitter (figure 3.11). The emitter has a single sound scene with a single channel. The player can change the volume of the emitter or the channel, change the active instrument and tempo and queue a motif. The ChangeCurrent toggle allows the player to decide whether the motif will interrupt the current fragment and whether the instrument change will play at the start of the next fragment or begin immediately by crossfading. Fig. 3.11: Legato_Pad demo scene. 22 Capítulo 3. The Toolset 3.6.2. Legato_Sample This scene, seen in the figure 3.12, is more similar to a demo of a game. The user can control a character represented by a blue sphere to move through the scene. Moving to a certain location may trigger an event: When the player enters the long corridor, the tempo will increment. Once out of it, the tempo will set to its initial value. When the player approaches the green character (at the bottom-right corner of the scene), a motif associated with this character will start playing. When the player enters the red room the instruments playing will change using a crossfade. Once the player is back to the white room, the instrument will change back (this time with no crossfade). Fig. 3.12: Legato_Sample demo scene. Cap´ ıtulo 4 Development The development of this project is split into three main parts: automating the rendering in Reaper, establishing the communication between Unity and Reaper and building the adaptive music logic and interface in Unity. 4.1. Automated rendering Reaper is the application chosen for rendering the motif samples from the composer’s MIDI items. To quickly recap the features of Reaper that makes it a great fit for the project: Reaper has a wide range of actions, from simple play and stop actions to the selection of individual tracks or creation of time selections. This list can be further expanded using Custom Actions and Reascript. Custom Actions are compound actions created by combining existing actions. Reascript is a scripting language that allows us to create actions from code (other actions can be invoked from this code). It has support for the programming languages EEL2, Lua and Python. The goal of this part of the project is to allow the composer to create a Reaper project (it is important that the composer does not need to use anything but the DAW) where they could create their instruments and melodies without having to go through the tedium of manually rendering each necessary combination. To automate the rendering process, a new action (Legato/render_motif.lua) has been created using Reascript. This script is written in Lua, making use of the ReaScript API (CockOS, 2024). It also makes use of many default Reaper actions, replicating the step-by-step process that a user would otherwise have to follow in order to perform the job manually. As mentioned in the previous chapter, in order for this script to function correctly, the Reaper project must obey the following structure: First, all of the tracks containing the instruments and effects. Then, the tracks containing the leitmotifs as MIDI items. Each track can have multiple MIDI items, which will be rendered together as one motif. 23 24 Capítulo 4. Development The Reaper actions used by the custom action cannot manage track numbers above 100, which is why the limit for the number of instruments and motifs cannot surpass 100 when added together. Reaper actions can only receive parameters by making use of pop-up windows with additional fields. Since the goal is to automate the entire procedure, the created action cannot ask the user for any parameters. Instead, an auxiliary plain text file is used to communicate the specifications between Unity and Reaper. This file is written automatically by Unity whenever a change to the configuration window is saved. The file has the following structure: <number of instruments> <list of instrument names, one per line> <number of motifs> <list of motif names, one per line> <number of tempos> <list of tempos, one per line> <instrument rendering vector for motif 0> <tempo rendering vectors for motif 0> <instrument rendering vector for motif 1> <tempo rendering vectors for motif 1> . . . The instrument and tempo rendering vectors are a row of 1s and 0s indicating, for each motif, whether it should be rendered in each instrument and tempo. The custom action starts by reading this file. Then, for each motif-instrumenttempo combination that needs to be rendered, the action performs the following steps: 1. Copy all the MIDI items from the track corresponding to the motif and pastes them in the track corresponding to the instrument. 2. Silence all the tracks except the one corresponding to the instrument (using Reaper’s solo function). 3. Adjust the project BPM to match the desired tempo. 4. Create a time selection that spans all the pasted MIDI items, then extend this time selection by an additional two beats. This allows most reverb, echo and release effects to finish without being cut off by the end of the sample. 5. Create a region that spans the same time as the time selection. 4.2. Cross-application communication 25 6. Queue a render of the project in the created region and time selection. 7. Delete the copies of the MIDI items, as well as the region and the time selection and un-solo the instrument track. Once all the combinations have been processed, the whole render queue is rendered. The files generated are left in the Assets/Resources/Legato/RenderedSamples directory in the Unity project. 4.2. Cross-application communication The auxiliary file explained in the previous section is written by the configuration window in Unity each time changes are saved. This window is made using the Unity’s EditorWindow API. Once the auxiliary file is written, all that remains is to notify Reaper to run the custom action. This is done using OSC, a network protocol for sound synthesizers, which has native support in Reaper. Reaper actions can be invoked via an OSC message using the action’s ID. This, however, presents a problem: custom actions created by Reascript have an ID generated when Reaper is opened. This means that the same custom action will have a different ID on each computer it is downloaded on. To circumvent this issue, an ID can be manually assigned to non-native actions in the file reaper-kb.ini (located within Reaper’s resources folder). A version of this file, edited to include a fixed ID for the custom action, is included alongside the action itself and the OSC listening device in the file ReaperConfigZip, included in the Unity package. This simplifies the process of configuring Reaper to work with Legato; the user only needs to import the ReaperConfigZip configuration file from Reaper. Unity does not have native support for sending OSC messages. The package extOSC (Sigalkin, 2016), from the Unity Asset Store, is used to add this functionality. extOSC provides, among other things, a pair of components for sending and receiving OSC messages from Unity. By making use of this package, the configuration window only needs to create a Transmitter component, using the action ID defined in reaper-kb.ini and the IP address and destination socket defined in the tool’s code. 4.3. Asset creation The three custom asset types used in this tool (Instrument,Tempo and Fragment) inherit from Unity’s ScriptableObject class. This class privides the memory-efficient storage of data that will be used or referenced many times in a project. It also grants a better user experience, as explained in the previous chapter. A progress bar displays the percentage of samples that have been rendered. After rendering all of the audio samples (which is detected by counting the number of files within the Assets/Resources/Legato/RenderedSamples folder), the window forces Unity to refresh its asset database. This ensures that Unity generates the correct 32 Capítulo 5. Conclusions and Future Work Using this tool, only the composer of a game needs to interact with the music generation software. The rest of the designers and developers can work with the intermediary assets that the tool generates for Unity. However, the composer does need to at least have access to the Unity editor. This is due to the dependency that the automated rendering process has upon the configuration window, located in Unity. Place as few restrictions as possible on the music and effects that the composer can create for the game. Many of the creative opportunities that Legato does not directly support can be facilitated through the use of Unity’s audio mixers, which the tool is fully compatible with. Even so, there remain some notable extensions that were not implemented (these are explored further in the Future Work section). Be easy to use. While this is a subjective point, the user interface is kept clean and somewhat minimalist, both in the visual editor and in the code’s public functions. A user doesn not need to know how the tool works internally in order to make use of it. The tool also does not intrude heavily on the development of a game. The set-up of a scene’s music is confined to a single Emitter and its auxiliary components. This frees the rest of the game’s code from being full of references to the soundtrack. Future Work While the tool is complete and functional, there is still plenty of room for expansion. This section explores several features that could be implemented upon the current version of Legato. Some of these were scrapped ideas from early in the project’s development, while others arose too late to be feasibly included for the deadline. 1. Currently, changes in tempo always take effect at the start of the next measure. The option could be added to allow the user to delay these changes until the start of the next motif (as is the case with instrument and motif changes). 2. Also concerning tempos, it was initially conceived that the project could play music at intermediate tempos between the ones rendered. This would require the use of pitch modification and interpolating between two rendered tempos to allow for gentle fades between tempos and a much greater degree of freedom with how these tempos are used. 3. All of the music that Legato plays originates from a single Emitter. Making use of Unity’s 3D audio features, an intriguing possibility involves allowing different parts of the soundtrack to have different positional settings. This would allow for games where many different characters have their own music that follows them around, all of it quantised in sync. 33 4. Regarding the Reaper project, the simplest improvements would be allowing the user to choose what audio format they would like to work with (currently, only .wav files are supported) and how long of a tail to add to the end of each rendered sample. This tail is the empty stretch of MIDI data that is added on the end of each sample, allowing long echoes, reverb effects and the like to end without being cut off. Currently, this is locked to always be equal to two beats of the sample’s tempo. 5. Allow Unity to be closed while the samples are rendered (to save resources and speed up the process) and opened once the process has finished. 6. Making use of the loadState property of Unity’s AudioSource class to circumvent the need for a loading buffer. This would also allow the tool to work more smoothly on extremely slow devices, while being more responsive on faster devices. 7. Detect changes in the Reaper project and use this information to avoid rendering samples that have not changed since the last render. 8. The user experience could be improved by an automated cross-referencing system that would detect contradictions between the configuration menu and the Reaper project, such as a mismatched number of instruments and instrument channels. In the case of such a contradiction, the user could be notified of the issue, rather than proceeding with a render that is already doomed to fail. Finally, the following expansion was proposed at the very start of the project, but was immediately discarded due to amount of work it would have required (it could potentially be a research project on its own). This would be the inclusion of advanced algorithms or artificial intelligence to analyse the leitmotifs provided by the composer and determine when they would sound best within the soundtrack. This way, a motif’s priority could be recalculated in every single measure to ensure that it plays at the best-sounding moment, creating the best possible harmony with the rest of the music. Introducción Introducción a la materia. Este capítulo contiene la traducción del capítulo 1. “Si hablas con cualquier director, te dirá que la música es el cincuenta por ciento de la película.” — Hans Zimmer Motivación Los hermanos Lumière contaban, ya en sus primeras proyecciones, con un pianista que acompañaba sus cortas películas (Tudor, 2017). Anterior incluso a la introducción del diálogo, la banda sonora ha sido siempre un elemento imprescindible del cine. Es difícil pensar en los protagonistas más célebres del cine sin recordar la música que anunciaba sus presencias. Don Corleone, con su trágica melodía de violín; la retumbante marcha militar del Imperio Galáctico; las tres instrumentos que definen a los epónimos protagonistas de “El Bueno, el Feo y el Malo”, a pesar de tocar todas las mismas dos notas(Classic FM, 2014); el elenco del Spider-verso, dotados de vida por la multitud de técnicas, instrumentos y aves de estanque que enredó Daniel Pemberton para esas películas (Sony Soundtracks, 2023). La identidad de estos personajes es casi tan musical como visual. Muchos de estos famosos sonidos son leitmotifs, figuras o frases melódicas asociadas a la aparición repetida de un personaje, idea o situación. Otros, como “El Bueno, el Feo y el Malo”, usan instrumentos para identificar a sus personajes. Este concepto de “identidad musical” nos fascinaba. Como estudiantes de desarrollo de videojuegos, queríamos una forma de emplear estas técnicas en nuestros juegos. La banda sonora de un videojuego tiene similar poder cautivador que la de una película; sin embargo, exceptuando los juegos de ritmo, la secuencia de eventos que ocurren en la mayoría de juegos no es determinista, pues se ven afectadas por eventos aleatorios y las decisiones de uno o varios jugadores. Por este motivo, es imposible componer canciones que resuenen perfectamente con cada momento de la acción. La música adaptativa surgió para remendar esta debilidad: una serie de técnicas de edición y reproducción que alteran la música del juego en el momento para ajustarse mejor a las acciones en pantalla. La música puede verse alterada con 35 36 Capítulo 5. Conclusions and Future Work cambios de instrumentación, ajustes de tiempo (velocidad a la que se reproduce la música) o introducción de nuevas melodías en la canción actual. Teníamos una imagen mental muy clara (o, más bien, un sonido mental) de lo que queríamos: una banda sonora de juego que describe mediante lenguaje musical la acción que está ocurriendo en pantalla, combinando leitmotifs e instrumentos identificativos de los personajes en escena. Como desarrolladores de videojuegos, buscamos middleware (es decir, software que funciona en una capa intermedia entre el código del juego y el sistema operativo) que nos permitiera crear música para videojuegos usando estas técnicas. Por desgracia, no pudimos encontrar una herramienta que permitiese hacer justo esto. Los compositores y desarrolladores usan herramientas como DAWs y trackers (dos tipos de aplicaciones usadas para producir música) para crear e integrar las bandas sonoras de sus juegos. Sin embargo, estas aplicaciones siguen requiriendo intervención humana y no necesariamente pueden ser controlados desde el motor de juego. Esta es una pega importante, pues la lógica del juego se programa desde el entorno del motor; sería poco manejable tener que editar los archivos fuente de música cada vez que un cambio en el juego afecte cómo y cuándo estos se reproduzcan. Sin una herramienta especializada, sería difícil implementar la funcionalidad que queremos. Cada combinación posible de leitmotif e instrumento necesitaría ser renderizada individualmente en cada tempo. Incluso en un entorno especializado, como una DAW (una aplicación usada para producir música), esta sería una tarea tediosa si no se usa ninguna automatización. Además, sincronizar una banda sonora formada por varias piezas dispersas sería una tarea difícil sin crear un sistema centralizado que lo maneje. Todas estas consideraciones nos ayudaron a fijar los objetivos del trabajo: crear un sistema de música adaptativa que utiliza técnicas musicales propias del cine, dando la posibilidad de utilizar instrumentos y efectos complicados o de alto coste sin requerir un alto coste de procesado. Definiciones clave Re-secuenciación horizontal: conjunto de técnicas de música adaptativa en la que se reproducen diferentes fragmentos de audio uno tras otro, cambiando el orden o la selección de fragmentos en respuesta a los eventos del juego. Re-orquestación vertical: conjunto de técnicas de música adaptativa que consisten en cambiar las partes individuales de una banda sonora que suenan de forma simultánea, alterando así la mezcla final. MIDI (Musical Instrument Digital Interface): formato de codificación digital de música para comunicación entre aplicaciones o dispositivos. Los objetos MIDI por sí solos no almacenan información del timbre del sonido; tan solo información de qué nota se toca, cuándo se pulsa y se suelta y algunos datos más sobre la forma en la que se pulsó. DAW (Digital Audio Workspace): dispositivo electrónico o aplicación soft- 37 ware usado para grabar, editar y producir audio. Entre otras cosas, permite renderizar muestras de audio a partir de una codificación (como MIDI) y una lista de efectos que combinan para determinar qué sonidos se producen a partir de cada nota. Instrumento virtual: software para generar sonido a partir de una representación simbólica (en el ámbito de este trabajo, generalmente en estándar MIDI). Estos instrumentos funcionan como plug-ins en una DAW. OSC (Open Sound Control): protocolo para la comunicación red multimedia entre dispositivos de audio. Crossfade: cambiar de un sonido a otro mediante un fundido gradual, entrando uno a la vez que sale el anterior. Cuantización: en el contexto de la música adaptativa, proceso de corrección y sincronización de diferentes fuentes de música para que suenen perfectamente en su tempo adecuado. BPM (Beats Per Minute): unidad para medir el tempo en música, equivale al número de pulsaciones en un minuto. Audio posicional: en los videojuegos, aquellos sonidos que simulan provenir de una ubicación concreta en el espacio. Objetivos El objetivo de este proyecto es crear una extensión para Unity que incluya herramientas para la inclusión de música adaptativa en videojuegos, usando las técnicas mencionadas anteriormente. Unity es un motor de videojuegos muy usado en el sector. Fue elegido para este proyecto principalmente por nuestra familiaridad con él, pero cuenta con otras ventajas. Dado que la herramienta está construida con y para Unity, está también orientada a los desarrolladores de Unity. La herramienta debe simplificar el proceso de crear una identidad musical para los diferentes personajes, lugares y temas dentro de la narrativa del juego, alterando la música ambiente cuando esos elementos narrativos entren en escena. Aunque la propia composición todavía debe ocurrir dentro de una DAW, los desarrolladores deberían poder hacer cualquier cambio a la lógica de cómo se reproduce la banda sonora sin tener que salir del motor de videojuegos. Estas herramientas deben: Facilitar la inclusión modular de leitmotifs en la banda sonora de un juego. Permitir cambiar en tiempo de ejecución la instrumentación de partes individuales de la banda sonora. Permitir cambiar en tiempo de ejecución el tempo de la banda sonora. 38 Capítulo 5. Conclusions and Future Work Ejercer poca demanda de recursos al ordenador del jugador. No exigir conocimientos de programación a los compositores. No exigir conocimientos de software musical a los programadores. Imponer la mínima cantidad de restricciones sobre la libertad creativa del compositor. Ser fácil de usar. Plan de trabajo Para lograr estos objetivos, el trabajo fue dividido en los siguientes bloques, algunos de los cuales se desarrollaron en paralelo: Investigación de tecnologías existentes sobre las que construir la herramienta. Para no escribir todo el programa desde cero, se estudiaron diferentes aplicaciones para evaluar su uso como base sobre la cual construir la herramienta. Esto también ayuda a enlazarla con un entorno con el que los usuarios ya estarían familiarizados, lo cual mejora la accesibilidad. Automatización del renderizado de archivos de audio. Dado que la herramienta usa muestras pre-renderizadas, estas tendrán que ser creadas automáticamente en función de ciertos parámetros definidos por el usuario. Creación de la interfaz de usuario para la configuración de la herramienta. El desarrollador debe tener acceso a un panel de control dentro del motor de videojuegos. Creación de los recursos de juego que representan los elementos dinámicos de la banda sonora. Para hacer que la herramienta sea más visual e intuitiva, los leitmotifs, instrumentos y tempos dinámicos serán representados como recursos dentro del motor de juego. Estos recursos tendrán que ser regenerados cada vez que se rendericen nuevas muestras de audio. Programación de la lógica de reproducción dinámica. El sistema de reproducción adaptativa es el corazón de la herramienta. Más allá de programar las entradas y salidas de los elementos dinámicos, los usuarios de la herramienta no tienes que gestionar la sincronización, la cuantización ni la reproducción de la música. Conclusiones y Trabajo Futuro Este capítulo contiene la traducción del capítulo 5. Conclusiones El resultado de este trabajo es una extensión de middleware para Unity, un motor de videojuegos con uso muy extenso, que facilita de creación de bandas sonoras adaptativas. Esto cumple con el objetivo principal. En cuanto al resto de pautas definidas en la introducción: Facilitar la inclusión modular de leitmotifs en la banda sonora de un juego. Permitir cambiar en tiempo de ejecución la instrumentación de partes individuales de la banda sonora. Permitir cambiar en tiempo de ejecución el tempo de la banda sonora. La inclusión de leitmotifs en la banda sonora es la principal funcionalidad de la herramienta. Con tan solo una llamada se puede insertar un leitmotif en tiempo de juego y estos suenan sincronizados entre sí. Cambiar la instrumentación o el tempo de estos leitmotifs es igual de sencillo, pues la herramienta los plantea como tres aspectos independientes de la música; se puede cambiar una cosa sin afectar las otras dos. Ejercer poca demanda de recursos al ordenador del jugador. Utilizar muestras de audio pre-renderizadas implica un menor coste en memoria que renderizar en tiempo de juego. En este aspecto la herramienta es comparable a alternativas como la que ofrece FMOD. Por otro lado, dejar al usuario elegir qué combinaciones de motif/instrumento y motif/tempo renderizar da opción a reducir el coste de almacenamiento ya que el usuario puede decidir guardar solo las muestras que va a utilizar. No exigir conocimientos de programación a los compositores. No exigir conocimientos de software musical a los programadores. 39 40 Capítulo 5. Conclusions and Future Work Usando esta herramienta, tan solo el compositor de un juego tiene por qué tocar software de audio. El resto de diseñadores y desarrolladores pueden trabajar con los recursos de Unity que la herramienta genera. No obstante, el compositor necesita al menos tener acceso al editor de Unity. Esto se debe a la dependencia que hay entre el renderizado automático y la ventana de configuración, que se encuentra en Unity. Imponer la mínima cantidad de restricciones sobre la libertad creativa del compositor. Algunas de las posibilidades creativas que Legato no proporciona pueden ser remediadas con el uso de los mezcladores de Unity. Aún así, hay algunas funcionalidades que no se han proporcionado (que se explorarán en la sección de Trabajo Futuro). Ser fácil de usar. Esto es un punto subjetivo, sin embargo sí que se ha cuidado la interfaz pública para que sea intuitiva, tanto la parte accesible desde la vista de editor como las llamadas por código. El usuario no tiene que entender cómo funciona la herramienta por dentro para poder hacer uso de ella. La herramienta es también bastante poco intrusiva sobre el desarrollo de un juego. La configuración de la música está muy encapsulada en el Emitter y sus componentes auxiliares. El código del juego no estará sobrecargado con llamadas a funciones referentes a la música. Trabajo Futuro Si bien el trabajo realizado da lugar a una herramienta terminada, esta se puede expandir de diferentes formas. A continuación se expondrán ideas de funcionalidades que pueden completar Legato. Algunas de estas ideas se plantearon al comenzar la investigación de este trabajo mientras que otras fueron surgiendo en la fase de desarrollo. 1. Los cambios de tempo se realizan siempre instantánemante, pero se podría dar la opción al usuario de retrasarlos hasta que cambie de motivo (como se permite en la reproducción de motivos o en los cambios de instrumento). 2. En relación también con los tempos, se planteó en un inicio permitir la reproducción de tempos intermedios a los definidos al renderizar. Hacer uso de atributos como pitch para modificar el tempo de un motivo en ejecución y dar total libertad en lo que a tempos se refiere. 3. La reproducción de música utilizando Legato está planteada para que haya un único Emitter en la escena de Unity del que proceda el sonido. Al exponer la funcionalidad de sonido 3D proporcionada por los AudioSource de Unity, sería interesante un replanteamiento donde se permitieran varios emisores y 41 un master, de forma que el sonido viniera de distintos puntos de la escena 3D. Esto podría dar lugar a juegos donde, por ejemplo, los motivos de dos personajes estén sonando a la vez pero cada uno desde un punto de la escena distinto, o donde haya una lista de motivos relacionados con la escena sonando en 2D mientras suena un motivo 3D focalizado en un punto. 4. Por parte del proyecto de Reaper y la automatización del renderizado, la primera idea (y la más simple) sería la de permitir distintos formatos para los archivos de audio (actualmente solo se renderiza en .wav). Además, se podría permitir al usuario definir el tiempo que se introduce al final de los objetos MIDI para no cortar los efectos. De esta forma, aún introduciendo efectos que alarguen mucho las notas (un eco muy prolongado en el tiempo, por ejemplo) no se apreciarían cortes. 5. Permitir que la ventana de Unity se cierre durante el proceso de renderizado de fragmentos (para ahorrar recursos y acelerar dicho proceso) y se vuelva a abrir una vez todas las muestras hubieran sido renderizadas. 6. Hacer uso de la propiedad loadState de la clase AudioSource para eliminar la necesidad de un tiempo fijo para cargar las muestras de audio. Esto permitiría poner menos presión a dispositivos muy lentos a la vez que permitiría exprimir más el rendimiento de los dispositivos de alta gama. 7. Detectar cambios en el proyecto de Reaper y usar esta información para evitar volver a renderizar muestras que no han sido modificadas desde el último renderizado. 8. La experiencia de usuario se vería mejorada si hubiera una comprobación de concordancia entre los datos introducidos en Unity (y trasladados al fichero intermedio) y el archivo de Reaper con sus pistas. Si el número de instrumentos o de motivos (o de pistas en general) no coincidiera con los valores introducidos en Unity, se informaría al usuario en lugar de intentar renderizar algo que, inevitablemente, va a dar lugar a error. Por úlitmo, esta siguiente idea se planteó al definir el trabajo, pero se descartó inmediatamente por su elevado coste de implementación (ya que podría suponer por sí misma un TFG). Sería el hacer uso de algoritmos que analicen los motivos para entender qué motivos armonizan bien entre ellos. De esta forma, el lanzamiento de motivos en ejecución no tendría por qué hacerse en el siguiente compás o al final del motivo actual, sino en el momento más oportuno para mantener la armonía y que el resultado sea el mejor posible. 48 Apéndice A. Legato User Manual The remaining tracks must correspond one-to-one with the leitmotifs that will be used in the soundtrack. These tracks can contain effects, but these will not appear anywhere in the final renders. Both the instrument tracks and the motif tracks must appear in the same order that the instruments and motifs do in the configuration menu in Unity. When the project is rendered, the MIDI items in the leitmotif tracks above will be combined with the individual instrument tracks to create the final audio clips. Some additional points: Calling the action directly from Reaper can cause problems, as any previous versions of the files created by the action will not be deleted. To avoid this issue, the action should only be called from the configuration menu in Unity. The beats per minute and beats per measure of the Reaper project will be overwritten. Reaper uses a render queue to store all individual renders before producing them all. Since this action flushes the queue, any previous renders that may still be in the queue will also be processed. The action saves the Reaper project before doing anything else, but does not save it after it finishes. This means that Reaper will detect the project as dirty (containing unsaved changes) even if there are no net changes relative to the previous state of the project. This also means that if the user wishes to recover anything that was overwritten by the action (such as the project BPM) they can simply close Reaper and discard the unsaved changes. Selecting a track name before the custom action is called can cause issues during rendering. To avoid this, before rendering, click anywhere in the area to the right, highlighted in green in the following screenshot: A.3. Legato Configuration 49 A.3. Legato Configuration Legato is configured from Unity, in the LegatoConfiguration pop-up that can be found in Window->Legato within the editor. The first fields of the window are for listing and naming the instruments, leitmotifs and tempos that will be used in the soundtradk. The number of instruments and leitmotifs set in this window must be the same as the number of tracks in the Reaper project. The next section contains two matrices to choose the combinations of motif/instrument and motif/tempo to render. At the bottom of the window there are buttons to save changes, render and regenerate assets. This last button should only be clicked if any issue ocurred during the creation of the Unity assets (after rendering) or if the user decided to manually create their audio files (not using this tool). In case the user created their files, these must be located in Assets/Resources/Legato/RenderedSamples, and their names must be formatted as: motif.<motif name>.<instrument name>.<tempo>.wav (all of this before clicking the regenerate assets button). For example, the clip for the motif “chords A” with the instrument “piano” and the tempo 120 should be called motif.chords A.piano.120.wav A.4. Legato_Emitter Description: The Legato_Emitter class manages the audio playback for different sound scenes, allowing for detailed control over volume, spatial blend, and more. Legato_Emitter is the main Unity component of the toolset; the source of the adaptive soundtrack. It can be found in the Assets/Legato/Components directory. The emitter is a singleton with a publicly accessible instance. The user can add the component to a game object in the scene or instantiate the prefab found in Assets/Legato/Prefabs. Serialized Properties: bool debug Activates informative message logs about the music and events being played during runtime. float volume Controls the global volume of the music that is played. SoundScene[] soundScenes •float spatialBlend How affected by 3D spatialisation the scene is. 50 Apéndice A. Legato User Manual •SoundSettings3D soundSettings3D Controller for the AudioRollofMode, minDistance and maxDistance attributes of a 3D scene. •Channel[] channels ◦Fragment[] fragments Array of musical fragments that will be played by default. ◦SortingAlgorithms algorithm The algorithm that is used to arrange the fragments. ◦float volume Controls the channel’s volume. ◦Instrument instrument The instrument that the channel will be played in at the start. ◦Tempo tempo The tempo that the channel will be played in at the start. ◦int beatsPerMeasure Number of beats per measure, used for synchronising channels and changing between fragments at the right time. ◦bool playOnStart Whether the channel will begin playing at the start or not. Public Properties: AudioMixerGroup output Set whether the audio should play through an Audio Mixer first or directly through the Audio Listener. const float loadingBuffer = 0.4f Time (in seconds) that is allowed for audio clips to load between measures. Public Methods: public static Legato_Emitter GetInstance() Returns the singleton instance of the Legato_Emitter. public void SetVolume(int c, float v) Sets the volume of a specific channel of the current scene. •int c: The channel to set the volume to. •float v: The volume of the channel (0.0 to 1.0). public void SetGlobalVolume(float v) Sets the global volume of the emitter. •float v: The volume of the emitter (0.0 to 1.0). public void SetScene(int s) Sets the current scene. A.4. Legato_Emitter 51 •int s: The scene to set as current. public void SetSpatialBlend(float spatialBlend) Changes the spatial blend attribute of the current scene. •float spatialBlend: The value of spatial blend, where 0.0 is fully 2D and 1.0 is fully 3D. public void SetSpatialBlend(int s, float spatialBlend) Changes the spatial blend attribute of the specified scene. •int s: The scene to change the spatial blend to. •float spatialBlend: The value of spatial blend, where 0.0 is fully 2D and 1.0 is fully 3D. public void Set3DRolloffMode(AudioRolloffMode rolloffMode) Sets the rolloff mode of the current scene’s 3D audio source. •AudioRolloffMode rolloffMode: The new rolloff mode. public void Set3DMinDistance(float minDistance) Sets the min distance of current scene’s 3D audio source. •float minDistance: The new value of minDistance. public void Set3DMaxDistance(float maxDistance) Sets the max distance of current scene’s 3D audio source. •float maxDistance: The new value of maxDistance. public void PlayMotif(int c, string motif, bool interrupt = false, float priority = 1f, float priorityOverTime = 0f) Queue a motif to be played in a specific channel. •int c: The channel to queue the motif in. •string motif: The name of the motif to add to the queue. •bool interrupt: (Optional) If true, interrupts any currently playing motif in the channel. •float priority: (Optional) The initial priority of the motif. •float priorityOverTime: (Optional) The rate at which the motif’s priority increases or decreases over time. public void PlayMotif(int c, Fragment motif, bool interrupt = false, float priority = 1f, float priorityOverTime = 0f) Queue a motif to be played in a specific channel. •int c: The channel to queue the motif in. •Fragment motif: The motif to add to the queue. 52 Apéndice A. Legato User Manual •bool interrupt: (Optional) If true, interrupts any currently playing motif in the channel. •float priority: (Optional) The initial priority of the motif. •float priorityOverTime: (Optional) The rate at which the motif’s priority increases or decreases over time. public void CancelMotif(int c, string motif) Remove all instances of a given motif of a specific channel motif queue. •int c: The channel to look for the motif. •string motif: The name of the motif to cancel. public void CancelMotif(int c, Fragment motif) Remove all instances of a given motif of a specific channel motif queue. •int c: The channel to look for the motif. •Fragment motif: The motif to cancel. public void Resume(int c) Resume a specific channel of the current scene. •int c: The channel to resume. public void Stop(int c) Stop a specific channel of the current scene. •int c: The channel to stop. public void StopAll() Stop all the channels of the current scene. public void ResumeAll(bool onlyPreviouslyPlaying = true) Resumes play on all the channels of the current scene. •bool onlyPreviouslyPlaying: (Optional) If true, only channels that were playing before StopAll was called will be resumed. public void SetInstrument(int c, string instrument, bool changeCurrent = false, float fadeDuration = 0f) Set the instrument of a specific channel. •int c: The channel to set the instrument to. •string instrument: The name of the instrument to set. •bool changeCurrent: (Optional) Change the current fragment or not, where true changes the current fragment’s instrument and false waits until the next fragment. •float fadeDuration: (Optional) The length of the crossfade in seconds when changing the instrument. A.5. Event triggers 53 public void SetInstrument(int c, Instrument instrument, bool changeCurrent = false, float fadeDuration = 0f) Set the instrument of a specific channel. •int c: The channel to set the instrument to. •Instrument instrument: The instrument to set. •bool changeCurrent: (Optional) Change the current fragment or not, where true changes the current fragment’s instrument and false waits until the next fragment. •float fadeDuration: (Optional) The length of the crossfade in seconds when changing the instrument. public void SetTempo(int c, int tempo) Set the tempo of a specific channel of the current scene. •int c: The channel to set the tempo to. •int tempo: The tempo in bpm to set. public void SetTempo(int c, Tempo tempo) Set the tempo of a specific channel of the current scene. •int c: The channel to set the tempo to. •Tempo tempo: The tempo to set. public void SetTempo(int tempo) Set the tempo in all channels of the current scene. •int tempo: The tempo in bpm to set. public void SetTempo(Tempo tempo) Set the tempo in all channels of the current scene. •Tempo tempo: The tempo to set. A.5. Event triggers To complement the emitter, the package also includes five components that simplify the process of activating events. These are Legato_InstrumentEvent, Legato_MotifEvent,Legato_TempoEvent,Legato_StopEvent and Legato_ResumeEvent. These components have a single public virtual method Trigger, which serves to activate the associated instrument, motif or tempo or to stop or resume specific channels, respectively. These five classes inherit from the abstract class Legato_Event, allowing them to be used polymorphically. 54 Apéndice A. Legato User Manual A.5.1. Legato_Event Description: The Legato_Event class represents an abstract event in the Legato system that can be triggered to perform specific actions related to audio playback. Protected Properties: Legato_Emitter emitter The Legato emitter instance. int channel The audio channel that the event will trigger on. Public Methods: public abstract void Trigger() Abstract method to trigger the event. Subclasses must implement this method. A.5.2. Legato_ResumeEvent Description: The Legato_ResumeEvent class represents an event that triggers the resuming of audio playback in the Legato system. It inherits from the Legato_Event class. Public Properties: bool applyToAll If true, triggering this event will resume all channels at once. bool onlyResumePreviouslyPlaying If true, only channels that were playing before StopAll was called will be resumed. Public Methods: public override void Trigger() Triggers the event. If applyToAll is true, resumes all channels. Otherwise, resumes the specific channel. A.5.3. Legato_StopEvent Description: The Legato_StopEvent class represents an event that triggers the stopping of audio playback in the Legato system. It inherits from the Legato_Event class. Public Properties: bool applyToAll If true, triggering this event will stop all channels at once. A.5. Event triggers 55 Public Methods: public override void Trigger() Triggers the event. If applyToAll is true, stops all channels. Otherwise, stops the specific channel. A.5.4. Legato_MotifEvent Description: The Legato_MotifEvent class represents an event that triggers the playing of a motif in the Legato system. It inherits from the Legato_Event class. Public Properties: Fragment motif The motif that will play on triggering the event. bool interrupt If true, the motif will be played in the selected channel at the next possible bar. If false, it will be added to the dynamic priority queue. float priority Triggered motifs are added to a queue. When the current fragment ends, the highest priority motif will play next. float priorityOverTime Rate that the motif’s queued priority changes over time (in units/second). Can be positive or negative. If the motif’s priority is below 0, the motif is removed from the queue. Public Methods: public override void Trigger() Triggers the event, playing the motif on the specified channel with optional interruption and priority settings. A.5.5. Legato_InstrumentEvent Description: The Legato_InstrumentEvent class represents an event that triggers the setting of a new instrument in the Legato system. It inherits from the Legato_Event class. Public Properties: Instrument instrument The instrument that will be set on triggering the event. 56 Apéndice A. Legato User Manual bool changeCurrent If true, any music currently playing will be crossfaded to the instrument on event trigger. If false, the current fragment will finish playing with its current instrument and the next one in the channel will use the new instrument. float fadeDuration If greater than 0, triggering the event will cause the channel to crossfade its instrument over fadeDuration seconds. Public Methods: public override void Trigger() Triggers the event, setting the instrument on the specified channel with optional crossfade. A.5.6. Legato_TempoEvent Description: The Legato_TempoEvent class represents an event that triggers the setting of a new tempo in the Legato system. It inherits from the Legato_Event class. Public Properties: Tempo tempo The tempo that will be set on triggering the event. bool applyToAll If true, changes in tempo will apply to all channels at once, regardless of selected channel. Public Methods: public override void Trigger() Triggers the event, setting the tempo on the specified channel or globally with optional application to all channels.