Sonification with musical characteristics : a path guided by user engagement
Full text
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University ABSTRACT Sonification with musical characteristics can engage users, and this dynamic carries value as a mediator between data and human perception, analysis, and interpretation. A user engagement study has been designed to measure engagement levels from conditions within primarily melodic, rhythmic, and chordal contexts. This paper reports findings from the melodic portion of the study, and states the challenges of using musical characteristics in sonifications via the perspective of form and function – a long standing debate in Human-Computer Interaction. These results can guide the design of more complex sonifications of multivariable data suitable for real life use. 1. INTRODUCTION Sonifications that rely on musical elements such as rhythm, pitch, and harmony are often reserved for creative purposes, yet musical characteristics can offer important contributions towards analysis [1]. When a variety of musical elements are integrated in auditory display for analytical contexts, such as process monitoring, researchers can capture a variety of features from multivariate data [2]. One concern, however, is that musical representations can create distractions and that music adds something more than is needed for an analysis. In this sense, music is seen as a more favorable component of form rather than function – an issue tied to the long-standing Human-Computer Interaction (HCI) debate of which comes first [3]. This paper offers the view that musical elements can facilitate analytical objectives from improved data perception, data analysis, and data interpretation through user engagement and enhanced experiences. The use of musical elements and characteristics in sonification has been formally explored by members of the International Community for Auditory Display since the mid 1990’s [4], [5], [6]. A summary of music-related research and software development in sonification can be obtained from Bearman and Brown [7]. Earlier attempts were made by Pollack and Flicks in 1954 [8], yet according to Paul Vickers [9] and other researchers, such as, Walker and Nees [10], the path for music in sonification remains uncertain or unclear. Vickers recently raised the significant question: “how should sonification designers who wish their work to be more ‘musical’ approach the task?” Vickers asserts “the path we ought to take is currently unanswered.” Walker and Nees [10] say that “questions of aesthetics and musicality remain open in the field of sonification.” The interest in incorporating musical characteristics in sonification can be summarized from the research of Brown et al. [11]: “the use of musical sounds has been recommended because of the ease with which musical sounds are perceived.” The use of musical characteristics in sonification can serve multiple purposes. To start, musical expressions that relate to aesthetics can enhance the user experience, and this can, in turn, enhance the perception of data. Positive user experiences can translate to more time spent with the data analysis, and improving interpretations. Finally, the mapping possibilities to musical traits are numerous and this brings opportunities to associate multivariate data to different types of musical features and expressions. The work reported in this paper was carried out in a project called Data-to-Music. It focused on the development of a software to map data to musical characteristics. The data came mostly from monitoring the conditions of buildings, machines, weather, and athletic experiences. What made the project unique, was the focus on the user experience with music. In order to test the possibilities of musical characteristics in data representation and display, the research team designed and programmed the software called D2M (Figure 1). It enables the users to assign SONIFICATION WITH MUSICAL CHARACTERISTICS: A PATH GUIDED BY USER ENGAGEMENT Jonathan Middleton1, Jaakko Hakulinen2, Katariina Tiitinen2, Juho Hella2, Tuuli Keskinen2, Pertti Huuskonen2, Juhani Linna2, Markku Turunen2, Mounia Ziat3 and Roope Raisamo2 1 Eastern Washington University, Department of Music, Cheney, WA 99203 USA, [email protected] 2 University of Tampere, Tampere Unit for Computer-Human Interaction (TAUCHI), Tampere, FI-33014 Finland, {firstname.lastname}@uta.fi 3Northern Michigan University, Dept. Psychological Science, Marquette, MI 49855 USA, [email protected] This work is licensed under Creative Commons Attribution – Non Commercial 4.0 International License. The full terms of the License are available at http://creativecommons.org/licenses/by-nc/4.0/ KWWSGRLRUJLFDG 35
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University variables of time series data sets to “streams” which control different aspects of the generated audio. Each stream can be associated with a unique or shared instrument timbre and the data values can be set to control pitch, the complexity of rhythmic patterns, accent chords, articulation, or loudness of the instrument. The user can determine the duration of the generated music and the data are scaled accordingly. The data streams specify parameters of each generated note based on the given settings. The global structure of the generated music is the result of the data content. Additional tools include the possibility to set thresholds based both on data values and time range to filter out some of the data. There are also preprocessing tools for linear and logarithmic scaling, as well as ways for inverting data values. The high degree of flexibility of the tool enabled the team to create auditory displays that sound composed, even though the results were prepared by an algorithm. In this article, we first outline our sonification approach and then describe a user survey that seeks to measure to which extent the approach leads to greater levels of user engagement. The D2M software was used to turn weather data into musical pieces used in the survey. 2. FORM AND FUNCTION The form and function debate in HCI is similar to the one found in sonification with music. In the process of interpreting data, sound characteristics with musical qualities can contribute to enhanced levels of engagement or distract from the analysis [12]. The challenge is to make sure the aesthetic qualities of the auditory display do not compromise coherence. We submit that data-to-music sonification can not only engage users, but levels of engagement with coherence can be increased by careful combinations of musical characteristics. A significant goal in HCI is the user experience, which is often tied to aesthetics. Dillon [13] traces the importance of aesthetics, as a basic motivator, back to A. Manslow (1954). Tractinsky et al. argue that aesthetic perceptions are highly correlated by individuals to interface usage; therefore, whether we intend to incorporate aesthetics by design into usage or not, aesthetics are most likely already tied to usage [3], [14]. In 2006 Andrew Dillon [13] introduced the term mediation to describe an essential mechanism whose role is to translate information between data and human. In describing this mechanism, he states that information technology is a mediator, among others, which “carries, stores, retrieves, and presents data”; and from these functions, information technology “can provide a physical instantiation of data or the mechanism for making visible or audible the data of interest.” In this paper, we are guided by the research question: Can musical characteristics contribute to meaningful data perception, analysis and interpretation? In addressing this challenge, we reference two studies that focus on the significance of the audible experience Data-2-Music 2.1 Pl a yb ac k contro l s C louds: stream 1 40 Pitch 6: marimba 40 60 Save settingsEdit Mute Mute 40 D uration ( seconds ) : Da t a to o n i n s tr u m e nt e ats / timeline L oo p : BPM : B eats / timeline: C louds weather Nov 20, 2016 - Nov 21, 2016 Logged in as admin. weatherLog out Reset timelines 2 PM 06 PM Nov 20 06 AM 12 PM 06 PM Mon 21 06 AM 12 PM 12:00 15:00 18:00 21:00 00:00 03:00 06:00 09:00 12:00 12:00 15:00 18:00 21:00 00:00 03:00 06:00 09:00 12:00 8 6 4 2 0 -4 -5 -6 -7 -8 -9 Play all Stop all Export MIDI Settings Settings Add stream Be P i t c h 43 60 MIDI R ange e nforce tonic - S cal e c-major Figure 1: Weather mapping to MIDI notes with the D2M software. 36
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University mentioned by Dillon and in particular the instantiation of data called sonification. We will then seek to prove that interpretive experiences with the aid of musical characteristics in the auditory display can offer analytical advantages. In Human-Computer Interaction, auditory display has maintained an increasing role in offering new perspectives on data analysis and interpretation. These processes are undoubtedly complex. Dillon simplifies the process with a figure using two key terms called arcs (or paths), similar to Don Norman’s interaction cycle [15]. There is the “arc of exploration” from human to data, and the “arc of interpretation” from data to human. Each path crosses the barrier of mediation (a central line) in a continuous circle. In other words, one leads to the other with the human as a starting and ending point. We suggest that the human experience within the arcs of exploration and interpretation can be enhanced by aspects of aesthetics that come with musical characteristics. A step in this direction is not easy and can be demonstrated by a recent study in condition monitoring. 3. SONIFICATION AND CONDITION MONITORING Condition monitoring with auditory display presents one of the most significant areas of exploration for sonification for practical usages. A recent study by Hildebrandt et al. [16] demonstrated the challenges of continuous monitoring and a successful method for real time peripheral monitoring with sounds. Their main goal was to create a continuous soundscape based on sonification that does not distract users from their main task(s). In their study forest sounds were chosen for continuous soundscapes during peripheral process monitoring. The decision to use sounds from nature came after consulting the work of Gaver et al. [17] and Vickers et al. [18] to achieve “unobtrusiveness” and avoid a sense of “disturbance.” One can imagine, in a practical sense, that sounds with musical characteristics would be more challenging to hear during extended “continuous” periods; however, given this assumption, in non-continuous environments, there may be opportunities for the inclusion of sonifications with musical characteristics. While their study was successful, their conclusion asserts: “potentially more pleasing sound designs will be tested to test the suitability for long periods and effectiveness.” They submit that these sounds could be based on longer looped samples “or even musical concepts.” Although forest sounds are often pleasant over long periods, the level of engagement may be quite uniform. The researchers’ motivation for more pleasing sound designs may be related to interests in creating a more engaging environment. 4. USER ENGAGEMENT Multiple studies have shown that the task of defining and measuring engagement is quite complex, and many evaluations are connected to the field of education to Pla y back controls C louds: stream 2 C louds: stream 2 40 Pitch Pitch 6: marimba 6: marimba 40 40 60 60 Mute Mute Mute 40 D uration ( seconds ) : D a t a to Data to o n in s tr u m e nt on in s tr u m e nt e line B eats / timeline L oo p : BPM : B PM: B eats / timeline: C louds 09:00 12:00 15:00 18:00 21:00 00:00 03:00 06:00 09:00 09:00 12:00 15:00 18:00 21:00 00:00 03:00 06:00 09:00 B B 0 -2 -4 -6 6 4 2 0 Play all Stop all Export MIDI Settings Settings Settings Preprocessing Add stream me R hythm sim p le com p le x v aryin g d ata t i mestamps 2 00:00 12:00 15:00 18:00 21:00 00:00 09:00 06:00 00:00 6 4 2 0 Figure 2: Cloud cover data being mapped into two independent streams of pitch and rhythmic values. 37
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University understand how students are engaged in learning [19], [20]. Engagement levels relative to musical interaction and loss of focus were recently explored by Tahiroglu et al. [21] where they monitored facial expressions of musicians and used the Positive and Negative Affect Schedule (PANAS) questionnaire to validate levels of perceived engagement. In various studies on engagement, common descriptive terms emerge, such as motivation, persistence, and effort. Lutz Klauda and Guthrie [20] elegantly separate motivation as goal-oriented based on values and beliefs, while engagement refers to “behavioral displays of effort, time, and persistence in attaining desired outcomes.” Additional attributes from other studies include focused attention, curiosity, novelty, and challenge [22], [23], [24], [25]. The most seminal work on engagement evaluation appears to be by O’Brien and Toms [26]. The O’Brien and Toms study defines engagement in great detail by building on the premise that engagement is a process in three stages: point of engagement, sustained engagement, and disengagement, and as the process unfolds in time, there are multiple layers of experience, called threads. The categorization of threads is derived from the work of McCarthy and Wright [27]. We surmise that musical characteristics can contribute to each thread: a) compositional (narrative), b) spatiotemporal, c) emotional, and d) sensual. While the O’Brien and Toms study is based on a framework for assessing levels of engagement in the context of visual display, our study adopts a similar framework to fit auditory/musical contexts. 5. USER SURVEY The impact of musical characteristics in sonification is being measured by a user engagement survey that comprises three 40 minute tests, each anchored by either melody, rhythm, or accent chords (as opposed to harmonic progressions). Each test relies on fundamental areas of musical expression: pitch, rhythm, and timbre. As an example, the Melody Test uses these musical elements to express weather forecasting data for clouds (called oktas); and by virtue of the use of inversion the data can also express amounts of sunshine perceived by listeners. Note: This inversion technique is similar to one described by Walker and Kramer [28]. 1 The oktas data, on a scale of 0-8, were mapped to a one and half octave range with a tempo of one beat per second. Forty data points in a 24 hour period were displayed in forty seconds. A MIDI range of 43-60 was used as a destination span and all results adhered to a C major diatonic collection (Figure 1). The resulting MIDI Temperature trends were perceived as pitches ascending to represent a rise in temperature (with heat), or inverted so that ascending 1 pitches represent cold, and although the latter seems counterintuitive, subjects perceived cold slightly better when pitches were high. Figure 3: Melody Test flow with cloud cover data from three separate months shown as: Data 1, Data 2, and Data 3. Participants listened to each condition twice with different tasks before completing the questionnaire.. 38
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University files were rendered through software instruments on Logic Pro X and Ableton Live, and the resulting audio files were exported and converted to mp3 files for Web delivery with loudness normalized to negative 25 LUFS. In our first test, listeners are presented with different experiences of three data sets in melodic form to determine if they hear clouds or sunshine. There are three conditions: Pitched sine waves with uniform rhythmic values with quarter notes (or crotchets); pitched timbre (marimba) with uniform rhythmic values with quarter notes; and pitched timbre (marimba) with a few basic rhythmic values in combinations using quarter notes and eighth notes (or quavers) (Figure 2). In the following, these three conditions are referred to as Sine, Timbre and Rhythm. Each melodic experience is associated with: 1) tasks with questions (to give users time to experience the sounds), and 2) survey questions, at the end of a set of tasks for a given condition. The participants complete the same user engagement scale (UES) questionnaire on three occasions, with questions modelled after O’Brien and Toms [29] and answered on a 5-step scale from “strongly disagree” to “strongly agree.” The participants complete tasks from three conditions (Figure 3) in two counterbalanced schemes of 6 (one scheme nested in another), hence participant 1 has the same experience as participant 7. Examples from our questionnaire below demonstrate how the Data-to-Music team adopted the original O’Brien and Toms [29] questions from a Web shopping experience for a musical context. “I was absorbed in my shopping task” transformed to “I was absorbed in my listening task.” “I felt frustrated while visiting this shopping website” changed to “I felt frustrated while listening to these sounds.” “This shopping website is attractive” became “These sounds were attractive.” The transformation of questions was as close as we could make them. In a few cases the transformations offered a challenge. For example, we modified the question “I liked the graphics and images used on this shopping website” to “I liked the beats and rhythms used in these sounds.” The team is satisfied with the transformations based on a pre-existing engagement study and how the user experiences from our sonifications may relate to varying levels of engagement. The complete set of modified questionnaire items can be seen in Table 1. We raffled the item order once and used the order (seen in Table 1) across all conditions and for all participants. Similarly as with O’Brien and Toms [29], we used reverse coding in the analysis for 8 out of 31 items for easier visual interpretation. Thus, " "& # #& $ $& % %& & "#&'#$ #& #' ) "" "& "' ## #( #) $" ( * "* #" #% $ "% ") #! #* % "! "# "$ "( $! Figure 4: Average UES responses. The lines represent the three conditions, Sine, Timbre and Rhythm, and the related data points represent the average responses to specific items (x-axis). The items are ordered by the UES factors. Items that revealed statistically significant differences in Friedman’s test (p<0.05) are marked as follows: *) some differences, but none identified in the pairwise comparisons between the conditions, SR) differences between Sine and Rhythm, and ***) differences between both Sine and Timbre, and Sine and Rhythm. 39
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University item 11 can be considered as “I did not feel frustrated while listening to these sounds.” The melodic portion of the study was conducted with 24 participants (11 female, 13 male) aged 18–50 years (M=22.4, SD=7.5). 17 of the participants had not heard data sonifications before the experiment. IRB exemption was provided by Eastern Washington University (human subjects protocol HS-5429) on December 1, 2017. 6. RESULTS AND CONCLUSION Results from 24 participants in our validation study indicate how combinations of musical characteristics in melodic forms can engage listeners, as these relate to the sonification of sun and clouds. The subjective results on the UES items per condition, i.e., Sine, Timbre or Rhythm, are represented in Figure 4. The figure shows average values instead of medians for easier visual distinction. We ran Friedman’s tests to compare whether the ordinally scaled data would show differences between the conditions. The analysis revealed altogether 19 items with statistically significant (p<0.05) differences, and 8 of these findings showed such differences even between the conditions in pairwise comparisons. Based on our results, engagement levels increase with melodies that are based on pitch with timbre combined, and those with pitch, timbre and rhythm combined. The findings show that sun and clouds data converted to only pitches with sine waves had lower engagement levels overall than those with timbre and rhythm (Figure 4). Average response values on a scale of 1–5 in the range of “strongly disagree” to “strongly agree” broadly demonstrate the importance of timbre in the six factors defined by O’Brien and Toms [29]. In “perceived usability” and “aesthetics” the differences are most pronounced. For the factor “perceived usability” averages are: sine waves 3.47; timbre and pitch 3.96; timbre pitch and rhythm 3.82. In the area of “aesthetics” averages are: sine waves 2.74; timbre and pitch 3.59; timbre pitch and rhythm 3.84. “Perceived usability” relates to the emotions experienced by users, and “aesthetics” to sensory appeal [29]. We see similar trends among some questions in the other factors: “focused attention,” “felt involvement,” “novelty,” and “endurability.” The trends suggest that timbre can significantly contribute towards engagement levels, even without the aid of moderately active rhythms. These results are from the Melody Test, which represents one third of our validation study. Tests with accent chords and active rhythms are on-going at this time. The conducted study provides understanding on the relevance of timbre and rhythm to the engagement with musical sonifications. Both of these properties appear to Table 1: The UES statements modified to address the context of sonifications. Item number indicates the order in which the statements were presented to the participants. * indicates that the item-related responses were reverse-coded. ' " ( " ) " * " + " ,! " - " . "# / " '& ! " '' "# '( " ') " '* " '+ " ', "# '- " '. " '/ " (& "# (' " (( "# () " (* " (+ " (, " (- "# (. "# (/ " )& " )' "# 40
The 24th International Conference on Auditory Display (ICAD 2018) June 10-15, 2018, Michigan Technological University have a significant effect, in particular to the aesthetics of the auralization. Aside from focused attention, other measured aspects are affected as well. While the evaluation was conducted with simple sonifications of single data variables to provide control to focus on individual factors on user experience, for real life use we believe that more complex sonifications are needed. Such sonifications can communicate more information and will likely have better endurability and more novelty. Our D2M tool supports the creation of these complex musical combinations. The observations presented in this paper set a foundation to investigate these more elaborate musical representations of data. 7. ACKNOWLEDGMENT Special thanks to Tekes, the Finnish agency for Innovation: decision 40296/14. This work was carried out in Finland in a three-year project funded by the agency and participating companies. The work was conducted at the University of Tampere within their unit for Human-Computer Interaction in collaboration with five companies with representatives seeking to hear their companies’ data for analysis. REFERENCES 1. Middleton J, Dowd D. Web-based algorithmic composition from extramusical resources. Leonardo 2008;41(2):128-135. doi:10.1162/leon.2008.41.2.128. 2. Hermann T, Hunt A, Neuhoff, JG. (Eds) The Sonification Handbook. Logos Verlag. 2011. 3. Tractinsky, N., Katz, A.S., & Ikar, D., What is beautiful is usable. Interacting with Computers, 13, 2000, pp. 127-145. 4. Hermann, T., Hunt, A., Neuhoff, J. G. (Eds), The Sonification Handbook, Logos Verlag, 2011. 5. Sound and Geographic Visualization. In Visualization in Moodern Cartography Mac Eachern and Taylor eds. New York: Pergamon, 149-166, 1994. 6. Flowers, Buhman, and Turnage. Data sonification from the desktop: Should sound be part of standard data analysis software?, in Proceedings of the 2nd ICAD 1996. 7. Bearman and Brown, Who’s Sonifying Data and How are they Doing it? A Comparison of ICAD and Other venues since 2009. Proc. 18th ICAD Atlanta GA., 2012. 8. Pollack, I. & Ficks, L., Information of elementary multidimensional auditory displays. The Journal of the Acoustical Society of America 26/155, 1954. 9. Vickers, P., Sonification and Music, Music and Sonification. In M. Cobussen, V. Meelberg, & B. Truax (Eds.), The Routledge Companion to Sounding Art (pp. 135–144). Oxford: Routledge, 2017. 10. Bruce N. Walker and Michael A. Nees. Theory of Sonification (chap 2 p. 27) in Sonification Handbook. 11. Brown, L.M. & Brewster, Stephen & Ramloll, S.A. & Burton, R & Riedel, B. (2003). Design Guidelines for Audio Presentation of Graphs and Tables. Proc. ICAD, Boston, MA, USA, July 6-9, 2003. 12. Vickers, P., Hogg, B., & Worrall, D., Aesthetics of sonification: Taking the subject-position. In C. Wöllner (Ed.), Body, Sound and Space in Music and Beyond: Multimodal Explorations, 2017, pp. 89–109. Routledge. 13. Dillon, A., Information Interactions: Bridging disciplines in the creation of new technologies. In Human-computer Interaction and Management Information Systems: Foundations. D. F. Galletta & Z. Ping (Eds.) Oxford: Routledge, 2006. 14. Tractinsky, N., Aesthetics in Information Technology: Motivation and Future Research Directions. In D. F. Galletta & Z. Ping (Eds.), Human-computer Interaction and Management Information Systems: Foundations. Oxford: Routledge, 2006. 15. Norman, D., The Design of Everyday Things, MIT Press, 1988. 16. Hildebrandt, T., T. Hermann, and Rinderle-Ma, S., Continuous sonification enhances adequacy of interactions in peripheral process monitoring. International Journal of Human-Computer Studies, 95, 2016, pp. 54-65, Elsevier. 17. Gaver, W.W., Smith, R.B., O'Shea, T., Effective sounds in complex systems: the ARKOLA simulation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Reaching Through Technology (CHI'91). ACM, New Orleans, 1991, pp. 85– 90. 18. Vickers, P., Laing, C., Fairfax, T., July 2014. Sonification of a network's self-organized criticality. arXiv:1407.4705. 19. Appleton, J. J., Christenson, S. L., Kim, D., Reshley, A. L., Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44/5, 2006, pp. 427-445. 20. Lutz Klauda, S., & Guthrie, J. T., Comparing relations of motivation, engagement, and achievement among struggling and advance adolescent readers. Read Writ, 28(2), 2015, pp. 239-269. 21. Tahiroglu, K., Vasquez, J. C. Kildal, J., Facilitating the musician’s engagement with new musical interfaces: Counteractions in music performance. Computer Music Journal, 41/2 2017, pp. 69-82. 22. Webster, J., & Ahuja, J.S., Enhancing the design of Web navigation systems: The influence of user disorientation on engagement and performance. MIS Quarterly, 30/3, 2006, pp. 661-678. 23. Webster, J., & Ho, H., Audience engagement in multimedia presentations. The DATA BASE for Advances in Information Systems, 28(2), 1997, pp. 63–77. 24. Jacques, R., Preece, J., and Carey, J. T., “Engagement as a Design Concept for Hypermedia, Canadian Journal of Educational Communications, 1995, pp. 49-59. 25. Skelly, T.C., Fries, K., Linnett, B., Nass, C., & Reeves, B. Seductive interfaces: Satisfying a mass audience. In C. Plaisant (Ed.), Proceedings of the Conference on Human Factors in Computing Systems, 1994, pp. 359–360. 26. O’Brien H. L., & Toms, E.G., What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 2008, pp. 938-955, Wiley. 27. McCarthy, J., & Wright, P., Technology as Experience. Cambridge, MA: MIT Press, 2004. 28. Walker, B.N. & Kramer, G. Mappings and Metaphors in Auditory Displays: An Experiment Assessment. ACM Transactions on Applied Perception 2/4, 2005, pp. 407-412. 29. O’Brien, H.L. & Toms, E.G., The Development and Evaluation of a Survey to Measure User Engagement: Journal of the American Society for Information Science and Technology, 61 (1), 2010, pp. 50-69. http://dx.doi.org/ 10.1002/asi.21229 41