Computer Music and Digital Media Art Through a Web-Based Collaborative Interface

MESTRADO MULTIMÉDIA - ESPECIALIZAÇÃO EM MÚSICA INTERATIVA E DESIGN DE SOM COMPUTER MUSIC AND DIGITAL MEDIA ART THROUGH A WEB-BASED COLLABORATIVE INTERFACE LUIS ARANDAS M 2019 FACULDADES PARTICIPANTES: FACULDADE DE ENGENHARIA FACULDADE DE BELAS ARTES FACULDADE DE CIÊNCIAS FACULDADE DE ECONOMIA FACULDADE DE LETRAS FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Computer Music and Digital Media Art through a Web-Based Collaborative Interface Luis Arandas Mestrado em Multimedia Orientador: Prof. Dr. José Alberto Gomes Co-Orientador: Prof. Dr. Rui Penha 17 de julho de 2019 c Luis Arandas, 2019 Computer Music and Digital Media Art through a Web-Based Collaborative Interface Luis Arandas Mestrado em Multimedia Aprovado em provas públicas pelo Júri: Presidente: Prof. Dr. Gilberto Bernardes Arguente: Prof. Dr. Rodrigo Carvalho Vogal: Prof. Dr. José Alberto Gomes 17 de julho de 2019 Resumo O presente documento apresenta uma investigação feita no contexto do Braga Media Arts, parte constituinte da rede de cidades criativas da UNESCO (UCCN). O objectivo e reflexão fundamental deste trabalho é dado na música por computador e na media arte a partir de interfaces colaborativas. Com um foco específico na performance artística e na interacção em sistemas digitais é proposto um ambiente audio-visual, Akson. Utilizando tecnologias web, o Akson foi inicialmente concebido como uma exploração do que poderia ser construído aproveitando a infraestrutura global da Internet, bem como a reprodução musical e visual em grande escala por vários dispositivos. É apresentada uma reflexão teórica da arte e da sua tecnologia enquanto veículo extensível da experiência, da pessoa e do sistema. Palavras Chave: Música por Computador, Media Arte, Arte em Rede, Performance Artística i ii Abstract This document presents a research made in the context of Braga Media Arts, part of UNESCO’s creative cities network (UCCN). The objective and fundamental reflection of this investigation is given in computer music and media art from collaborative interfaces. With a specific focus on artistic performance and interaction in digital systems, we propose an audio-visual environment, Akson. Using web technologies, Akson was initially conceived as an exploration of what could be built by leveraging global Internet’s infrastructure as well as large-scale musical and visual reproduction across multiple devices. We present a theoretical reflection of art and its technology as an extensible vehicle of experience, person and system. Keywords: Computer Music, Media Art, Networked Art, Artistic Performance iii iv List of Figures 2.1 Photo named "Cage. Variations" found in Arnolfini. An article about 9 evenings: Theater and Engineering............................... 8 2.2 Courtesy of the Authors. On the left a picture of Julien Bayle performing ALPHA v1.0 in Montréal and on the right a picture of Boris Divider on Data+Perform live performance. .................................... 10 2.3 On the left picture taken from Sónar 2018 official website of Carsten Nicolai (Alva Noto) performing and on the right picture taken from Ryoji Ikeda’s official website of the artwork text pattern. ............................. 11 2.4 Picture of the schematic diagram of a general communication system. Claude Shannon on The Mathematical Theory of Communication. ............ 13 2.5 Courtesy of RAND Corporation. Network Models proposed by Paul Baran in 1964. Image found in document RM-3420-PR. . . . . . . . . . . . . . . . . . . 14 2.6 Courtesy of the Authors. On the left a picture of Telesymphony performance and on the right a picture of BEER performing in Network Music Festival (2014) in Birmingham. .................................... 14 2.7 A picture of The League of Automatic Composers performing in San Francisco 1981 (McKinney, 2016; Brown, 2002) . . . . . . . . . . . . . . . . . . . . . . . 15 2.8 Picture found on article Long Distance Telephone Concerts by Scientific American, February 28, 1891, page 130. Performers in New York and audience in NewtonMassachusetts................................ 16 2.9 Courtesy of Alvaro Barbosa. A screenshot of FMOL software. A web-based treestructured database. c SergiJordà......................... 17 2.10 Kandinsky’s Yellow-Red-Blue (1925) is from his period of totally non-representational works, but this particular painting seems evocative of the concept of a person experiencing vivid ‘synesthetic’ imagery. Oil on canvas, Musée National d’Art Moderne, Centre Georges Pompidou. Reprinted with permission: Réunion des Musées Nationaux/Art Resource, NY. 2003 Artists Rights Society (ARS), New York/ ADAGP, Paris. Amy Ione and Christopher Tyler on Neurohistory and the Arts.......................................... 19 2.11 Image with both the official logo of Braga Media Arts (left) and the official logo from UNESCO’s UCCN (right). . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.12 Two screenshots of possible shapes that Shoggoth can graphically generate. These were found on Chad McKinney’s PhD document (2016) . . . . . . . . . . . . . 22 2.13 Picture on the left is a photograph of a performance in MIT media lab using GroupLoop on the right a print screen of the web interface found online. . . . . . . . . 23 2.14 Picture on the left photograph of the performance Fingerprints in Passos Manuel (Porto) in ICLI 2018. On the right a print screen of the editor used, Troop. . . . . 23 xi xii LIST OF FIGURES 2.15 On the left picture of the performance by Andrew Blanton on Soundwave and on the right a picture of the graphical content displayed on the screen throughout the performance. .................................... 24 2.16 An example of a distributed, synchronized musical instrument (DSMI). The instrument is composed of various parts that exist in different locations. It uses prediction in order to synthesize the same audio in each location at the same moment. It capitalizes on the predictable path of a percussion mallet in order to accomplish thisgoal........................................ 25 2.17 Picture on the left is a performance at the 2012 Network Music Festival in Birmingham (UK) with Yig and on the right is a screenshot of the software’s GUI. . . 25 2.18 On the left is a photograph of one of the performers at the concert presented at Casa da Música and on the right is a photograph of the public interacting via mobile at the concert held at Berklee. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.19 Picture on the left is a public performance of Monad with remote players in Istanbul, Vancouver and Berlin. On the right a screenshot of the software’s GUI. . . . 26 2.20 Two pictures of the Concert for Smartphones and Orchestra in Georgia Tech (2016) during Web Audio Conference (WAC). On the left Andrey Bundin on electronics and on the right as violin, Alexey Kotchetkov. . . . . . . . . . . . . . . . 27 3.1 Print Screen of Akson took by the authors. . . . . . . . . . . . . . . . . . . . . . 29 3.2 Courtesy of Publicações Europa-América. Image in O Erro de Descartes by António Damásio. Diagram of a neuron with it’s principal components: Cell Body, AxonandDendrites. ................................ 30 3.3 Starting on an open source code editor Atom (bottom-left) and then uploading it to the cloud through SourceTree (bottom-right) an open-source Git GUI software. When the system is fully updated we have the GitHub interface of the Akson repository with all the code commits that by itself interfaces directly with Heroku pipeline........................................ 31 3.4 On the left, screenshot of the offical readthedocs website. On the middle, screenshot of the commits tab in GitHub. On the right a screenshot of Akson’s bandcamp with music from public experiments. . . . . . . . . . . . . . . . . . . . . . . . . 32 3.5 Screenshots of the four graphical scenes of Akson in their default state. . . . . . 34 3.6 Picture of the Master Dialog (top-left), the System Dialog (top-center and topright) and a table with the various widgets and methods (bottom). . . . . . . . . . 37 3.7 In this figure we have 5 Akson print-screens. On the first (top) we have the default visualization of Akson. On the next 4 we have the shaders used in this work. They are instances of the EffectComposer and multiple Pass objects. On the middle row we have the GlitchPass shader with two modes of intensity (on the instruments panel called Glitch and Noise). On the bottom row we have on the left the afterimagePass, a method of graphic processing that creates kind of visual entrainment sometimes compared with reverb. And on the right column we have the Vignette effect that adds two black tabs to the main scene camera. . . . . . . . . . . . . . 39 3.8 Screenshot of the Synthesizer panel followed by two tables with the various widgetsfunctionsandmethods.............................. 45 3.9 Screenshot of the Background panel followed by two tables with the various widgetsfunctionsandmethods.............................. 46 3.10 Screenshot of the Graphics panel followed by one table with the various widgets functionsandmethods................................ 47 LIST OF FIGURES xiii 3.11 Screenshot of the Post-Prod panel followed by two tables with the various widgets functionsandmethods................................ 48 3.12 Screenshots of the Allocate dialog (top), the Logs dialog (middle-left), the Save Settings dialog (middle-right) and the Top bar with a panel containing methods of action......................................... 49 3.13 Screenshot of the modes of interaction modal triggered by the Top bar. . . . . . . 50 3.14 Keyboard map (QWERTY) with a panel containing methods of action, Keycode Events and the functions found in the code. . . . . . . . . . . . . . . . . . . . . 50 3.15 Diagram of Akson’s descenter model. ....................... 51 3.16 Diagram of Akson’s streamed model. ....................... 52 3.17 Diagram of Akson’s allocate model......................... 53 3.18 Diagram of Akson file tree and folder communication flow. Starting on the Procfile (top-left). ...................................... 54 4.1 A photograph that shows part of the space where the experiment was done, seven computers with Akson running and Joana Rodrigues, a professional musician and participant of this experiment. . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2 Courtesy of gnration’s communication manager. Photograph took on the site of the installation during the morning of open day................... 59 4.3 A photograph of the performance at Openfield. Courtesy of Marcelo Sousa, researcherandpublic.................................. 62 6.1 Synopsis made for the BMA Educational Service to be used as a sign in the space of the Akson installation on the open day. A paragraph is provided in English and in Portuguese to explain how you can interact with the work. . . . . . . . . . . . 69 6.2 Technical Rider developed for an interactive installation with Akson. This formalization as installation was thought for a more limited equipment load exploring in the same the visualization of the instrument. . . . . . . . . . . . . . . . . . . . . 70 6.3 Image shown to the participants before CARA’s experiment took place. Using this annotation it was possible for the participants to know how to interact minimally with the system. This image was available for viewing throughout the performance 71 6.4 Image shown to the participants on gnration’s interactive installation. This image was constantly present during the whole course of the installation. . . . . . . . . 72 6.5 Image shown to the participants on Openfield’s performance. The image was demonstrated to the participants just before the start and was not shown a second time.......................................... 73 6.6 Screen shots of Akson’s graphical system, taken from the virtual camera. These can be found in http://www.luisarandas.org/akson.html............... 74 6.7 Screen shots of Akson’s graphical system, taken from the virtual camera. These can be found in http://www.luisarandas.org/akson.html............... 75 7.1 Photograph of signage arranged at the entrance of the Akson installation in the open day. ...................................... 77 7.2 Article about Akson found on gnration flyer. . . . . . . . . . . . . . . . . . . . . 78 7.3 Article about Akson found on gnration website. . . . . . . . . . . . . . . . . . . 78 7.4 Courtesy of Ilídio Marques, gnration Communication manager. A high-resolution vertical poster image that was found across the city of Braga as publicity. . . . . 79 xiv LIST OF FIGURES 7.5 Courtesy of Ilídio Marques, gnration Communication manager. A high-resolution horizontal poster image of the event. Was also found across the city of Braga as publicity. ...................................... 80 7.6 Courtesy of Ilídio Marques, gnration Communication manager. A photograph of users interacting with Akson during the open day.................. 81 7.7 Courtesy of Marcelo Sousa, researcher and public at Openfield experiment. A photograph of the explanation given after the performance, maintaining an interaction with Akson using a mobile phone. . . . . . . . . . . . . . . . . . . . . . . 82 7.8 Article about Akson found on https://www.rimasebatidas.pt/gnration-celebra-sextoaniversario-com-open-day-recheado-de-musica-e-actividades/. A post by ReB TeaminFebruary22th................................ 83 7.9 A photograph of a gnration flyer containing the activities of open day. ...... 84 Abbreviations and Symbols 3D [Three Dimensions] ADSR [Attack Decay Sustain Release] AMD [Advanced Micro Devices] ANSI [American National Standards Institute] API [Application Programming Interface] AV [Audio-Visual] BEAST [Birmingham Electroacoustic Sound Theatre] BEER [Birmingham Ensemble for Electroacoustic Research] BMA [Braga Media Arts] CARA [Centro de Alto Rendimento Artístico] DAW [Digital Audio Workstation] DDR [Double Data Rate] DIP [Distributed Immersive Performance] DNS [Domain Name System] DOM [Document Object Model] DSMI [Distributed Synchronized Musical Instrument] DSP [Digital Signal Processing] ECC [Error-Correcting Code Memory] EMUI [Emotion UI] EQ [Equalization] ES [Embedded Systems] FFT [Fast Fourier Transform] FPS [Frames per Second] FOV [Field of View] GB [GigaByte] GHZ [GigaHertz] GL [Graphics Language] GLSL [OpenGL Shading Language] GUI [Graphical User Interface] HCI [Human-Computer Interaction] HD [High Definition] HDMI [High Definition Multimedia Interface] HTML [Hyper Text Markup Language] ICLI [International Conference on Live Interfaces] ICMC [International Computer Music Conference] INESC TEC [Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência] KHZ [KiloHertz] LFO [Low Frequency Oscillator] xv xvi ABBREVIATIONS AND SYMBOLS MB [MegaByte] MBPS [MegaBits per Second] MHZ [MegaHertz] MIDI [Musical Instrument Digital Interface] MIT [Massachusetts Institute of Technology] MP3 [MPEG-1/2 Audio Layer 3] MPEG [Moving Picture Experts Group] MS [Milliseconds] NIME [New Interfaces for Musical Expression] NMP [Networked Music Performance] OGG [Container Format by the Xiph.Org Foundation] OJM [Orquestra de Jazz de Matosinhos] OS [Operating System] OSC [Open Sound Control] OSX [Macintosh Operating System X (Apple Computer, Inc.)] PA [Public Address System] RAM [Random Access Memory] TRS ["Tip, Ring, Sleeve"Cable] TYPE [Yorkshire Programming Ensemble] UNESCO [United Nations Educational, Scientific and Cultural Organization] UCCN [UNESCO Creative Cities Network] VCS [Version Control System] VMM [Variable Markov Models] WAC Web Audio Conference WAV [WAVEform audio format] WI-FI [Wireless Fidelity] WWW [World Wide Web] XY [Coordinates of the Cartesian System] ZKM [Zentrum für Kunst und Medientechnologie] Chapter 1 Introduction This dissertation transmits the development and experimentation of a collaborative audiovisual (AV) environment done in the context of Braga Media Arts1, a recent creative city in the UNESCO’s network (UCCN)2. In 2017, Braga3was officially designated as Creative City of UNESCO in the field of Media Arts. In the recent past the city pointed to an increasingly technological future, reinventing its economic fabric focusing on technology companies, digital communication studios, laboratories and research centers in the field of robotics, games, multimedia and media arts. Simultaneously a cultural ecosystem has gained form and expression, defining itself in a contemporary and transdisciplinary identity, situated in the intersection between new media and formal artistic expressions. The use of technologies in artistic practice has led many projects to transport Braga to a dimension of international recognition. This research was done with a close relationship with its educational service, exploring the process of technological development and artistic experimentation of a proposed tool. Accompanied by historical and present references on networked interfaces, this document aims to show everything related to the project. The digital interface presented was done using web technologies exploring networked interaction in the performing arts. All software and interface design was developed specifically for this research and the entire technical development process exists in the cloud with free access. In this chapter we present the context and motivation that supported this research followed by the problems, hypotheses and objectives conveyed. Finally there is a description of the methodologies used and brief summary of the chapters comprised in the document. 1Official Braga Media Arts website http://www.bragamediaarts.com/en/ [accessed 27/06/2019] 2Official UCCN website https://en.unesco.org/creative-cities/home [accessed 27/06/2019] 3Braga is a city and a municipality in the northwest of Portugal. It is the third-largest urban center in Portugal (after Lisbon and Porto). 1 2Introduction 1.1 Context and Motivation The confluence of art and contemporary technology has always been a significant part of our work, both as artists and as critics. The consideration deposited in art and it’s technology has led us to formalize opinions about human interaction and aesthetics that are fundamental bases of the work presented here. The motivation of this work is rooted in influences from collaborative computer music to contemporary digital media art. Fundamentally we argue that systems like the one proposed are needed as artistic extension in the cloud, mainly as a way to do so. All of the people involved in this research (minus some audience) had clear musical, artistic, and academic backgrounds. The search for an action that corresponds both to the ethics of art and to the technological development of instruments equally is undoubtedly what defined the concept of the formalized environment. This work is also product of references and artistic choices of the authors that they have maintained for some years before this investigation. 1.2 Project The proposed project began with the establishment of a work flow that would allow a full year of software development without fail. A constant search for references in the course of the investigation was also maintained, always supported by the public artistic experiments. Our aim as researchers and artists is to integrate the computational environment developed in the performing arts and as a distributed digital collaborative interface. In this way it is possible to create a bond that is consistent with development and outcome. We develop and integrate in the cloud, an environment that can answer both the main research questions and our conceptual requirements. We are committed to developing it with web technologies creating interfaces for multiple devices. These technologies were chosen to allow digital multimodality both in the development of it as a tool and as an AV generation system. 1.3 Problem(s), Hypotheses and Objectives of Research Considering the massive worldwide development of web technologies, networked collaborative interfaces and digital AV generation, we define the various parameters of research. We can formulate our problem considering the following: •We have already extended our senses and our nerves by the various media (McLuhan, 1964) 1.3 Problem(s), Hypotheses and Objectives of Research 3 •The interactive capabilities opened up by computer networks allow for shared creativity, from textual or telematic real-time improvisations to globally constituted group projects, with distance no barrier to collaboration (Dixon, 2007) •New web standards such as Web Audio and Web GL introduce the capability for web browsers to duplicate many of the features in computer music tools (McKinney, 2014) •By default, the interface is a system or device through which unrelated entities can interact (Sá, 2010) Problem •How can we implement and validate a web-based AV tool for collaborative interaction? Hypothesis check •Can we, through the use of web-based technologies: –Develop a collaborative environment for AV networked interaction? –Use the internet as a vehicle for interaction between artists with potentially global distances around the world? –How can a multi user system for artistic creation mediate, and thereby enhance, the interaction between agents with distinct creative proficiencies? –How can a digital collaborative AV system contribute for artistic creation through the use of web-based technologies? –Validate the developed tool in public experiments using cloud tools? Objectives The main objective of this research is to develop a networked AV collaborative environment with web technologies Understand the role of a multi-user digital system as a mediation agent in computer music and AV performance. The proposal and validation of this tool should be implemented in a context of the Educative Service of Braga Media Arts. 4Introduction 1.4 Research Methodology In this sub-chapter we present the methodology and research design used. When this research was integrated in the context of Braga Media Arts, the objectives explained earlier were very clear, keeping the research questions defined during the work process. We, as researchers, have carried out a thorough review of the conceptual and technological literature that in turn extended to the end of the investigation. A balance of possible outcomes and similar projects/methodologies for art creation was always performed. The development process of the proposed environment (see chapter 3) was carried out with respect to practice-led research and research-led practice4(Smith, Dean 2009; Candy, 2006) methods and covering also the fundamental practices of artistic research5(Busch, 2009; Liamputtong & Rumbold, 2008) as an academic mode of production. After having a work flow defined and practicable in the scope of the research time scale we believed that the continuous technical development parallel to the artistic/theoretical research was the method to be adopted (computer-led practice). 1.5 Dissertation Structure This dissertation is organized in 5 chapters followed by the appendix, attachment and references. •The first chapter –In the first chapter we explore the foundations of this research, the type of project, the motivations, the resolution, the hypotheses and problems. •The second chapter –A review of the fundamental literature is made to instantiate our research in history, always accompanied with contemporary references. We present a sub chapter with recent works entitled state of the art. In here several important works made recently that are an important reference for this investigation are cited. •The third chapter 4A type of research that leads primarily to new understandings about practice. Is concerned with the nature of practice and leads to new knowledge that has operational significance for that practice (Candy, 2006) 5Fundamentally defined by art that understands itself as research, in that scientific processes or conclusions become the instrument of art and are used in the artworks. This refers to a particular phenomenon in contemporary art, in particular in institutional-critique, whereby research is considered a part of the artistic process and is carried out by the artist herself. In this case, art is in fact a form of knowledge (Busch, 2009) 2.1 Literature Review 11 2.1.4 New Media and Computer Graphics Always contributing to the drastic change in the creative processes with technological advances (Broad and Grierson, 2016) and having in mind that visual art might have a role to play in computer music performance, we form up a review on today’s digital computer graphics. In the preface of the book Cyclo. Id (2011) published by Ryoji Ikeda alongside with Carsten Nicolai (Alva Noto) we can read in the preface that the artists declare that they have “abandoned the idea that the image acts only as a functional accompanist to sound and instead subordinated the audio element for our desire of the image” (Ikeda and Noto 2011) (Carvalho, 2018). On this position before artistic practice we can see the graphical image having a different role to play inside the AV performance, not specifically an income or outcome to music but the original compositional matter. Figure 2.3: On the left picture taken from Sónar 2018 official website of Carsten Nicolai (Alva Noto) performing and on the right picture taken from Ryoji Ikeda’s official website of the artwork text pattern. Interesting work can also be found in the field of synaesthesia7and AV relationships. The work of Kandinsky on color/music (Kandinsky, 1947) and the Quayola’s sonic visualizers (Quayola, 2011a) are a big reference to take in account. Regarding the possibilities artists have on content generation, just like music there are several ways of making video or digital graphics with or without real-time interaction in artistic performance. Environments like Processing (Fry & Reas, 2001), OpenFrameworks (Lieberman, Watson, & Castro, 2005) (Gomes, 2014) a project started at MIT Media Lab conceived in 2001, or Cinder (Gómez, Colubri & Charalambos, 2016; Bell et al., 2012) a software that also has a good environment for the artist to program digital graphics and sound with the availability to use in their projects for any use (commercial or otherwise). 7Synaesthesia - Neurological Condition on Wikipedia https://en.wikipedia.org/wiki/Synesthesia [accessed 27/06/2019] 12 Contemporary and Historical Practice The massive development happening worldwide on the field of interactive graphics is astonishing and has great possibilities. Also, the development of visual computing nowadays has a big coherence with the type of hardware and API’s people have access to, the widespread availability of programmable graphics processing units (GPU’s) and shading languages such as GLSL (Rost et al., 2009) makes visual art easier to develop. Some media artists also use and exploit gaming programming systems, such as Unity8, Unreal Engine9or Godot10. 2.1.5 Networked Systems and Performance As humans, we create structures and networks for almost every type of collaborative behavior, and NMP has benefited from over three decades of development (McKinney, 2016). Examples of that can be the experiments of the San Francisco Bay Area network band pioneers and of course the development of the internet. Introduction of protocols like Open Sound Control (OSC) (Wright, 2002) for multi-purpose communication present good ways of extending those networks. There is also relevant work on the field of collaboration and networking that presents taxonomies and extended research that help us structure a better relationship with the past. Such examples are Displaced Soundscapes - Computer Supported Cooperative Work for Musical Applications (Barbosa, 2003) and Internet Music: An Introduction (Hugill, 2005). And to don’t lose definition, the term network music can quickly become opaque and without meaning. For that I would like to bring the book Networking Foundations (Ciccarelly & Faulkner, 2004) where a network is defined as a system that allows communication to occur between two people or machines (McKinney, 2016). When we talk about networked digital systems we are also thrown to the big infrastructure of those systems, and there is a problem raised by Bridle (2018) that studies the opacity of that computation. As digital systems take place inside the machine, hidden inside buildings and big structures that propagate themselves as long as wires and WI-FI enables, there is a barrier from the user. And even when we disrupt that barrier, it remains beyond the comprehension of most. The aggregation of complex systems in contemporary networked applications means that no single person ever sees the whole picture (Bridle, 2018). That is a problem to human interpersonal behaviour and brings us multiple sociological paradigms (e.g. automation bias) (Parasuraman & Manzey 2010), but might not be a problem in performatic interaction. The outcome of the piece is multiple times the most important thing, and if there is a clear trade of information in the main path of communication (path defined as sender-receiver link) with a controlled amount of noise (Shannon, 1948) that might be a problem solved (both for the public and performers). To the communication and to the embodiment. The 8Unity main website https://unity.com/ [accessed 27/06/2019] 9Unreal Engine main website https://www.unrealengine.com/ [accessed 27/06/2019] 10Godot main website https://godotengine.org/ [accessed 27/06/2019] 2.1 Literature Review 13 point that defines controlled communication as a scale is based on the model defined by Claude Shannon (1948). Figure 2.4: Picture of the schematic diagram of a general communication system. Claude Shannon on The Mathematical Theory of Communication. Nowadays, from the simplest social media user to the most complex serious information structure, we rely on an incredibly big communication field. That field by default, tries to erase this noise source and leaves us space to develop creative ideas and ways of experimenting it in new ways. These new ways of expressing can sometimes take place in an interactive distributed computer system for real-time music made by multiple users (Brown, 2007). To differentiate the models for collaborative behaviour as types, we have to split systems by data communication architectures. These networked models for collaborative actions might be divided mainly in Centralized Network Models,Decentralized Network Models, and Distributed Network Models (Hernst, Gurle, & Petit, 2000; Pulkka, 1995). These different ways of building software and flows of information also bring different ways of exploring it, as medium or the art itself (Blanton, 2019; McLuhan, 1967). And as there are new ways of exploring it, new ideas and systems come to life. A good example of musical performance is the iconic work Dialtones (A Telesymphony) (Bédat, 2002), a concert whose sounds are wholly produced through the carefully choreographed ringing of the audience’s own mobile phones, presented at Ars Electronica Festival (2001) and at the Swiss National Exposition (2002). In contrast to that model of interaction we have the The Birmingham Ensemble for Electroacoustic Research (BEER) performing at the Network Music Festival in 2014 with the three members sitting at a table. Surrounded by the audience contrasting 14 Contemporary and Historical Practice Figure 2.5: Courtesy of RAND Corporation. Network Models proposed by Paul Baran in 1964. Image found in document RM-3420-PR. the effort to show that the music is played in real-time, this performance explores a minimalist interaction with the computer (Akkermann, 2016). Figure 2.6: Courtesy of the Authors. On the left a picture of Telesymphony performance and on the right a picture of BEER performing in Network Music Festival (2014) in Birmingham. Complex networking schemes allow for the sharing of performance data between musicians, however, the connection between the music and the listeners – both ensemble participants and audience members can be diffuse and unfocused (Diakopoulos, 2012), that’s why I bring two different examples of performative interaction. It’s also important to notice techniques like live-coding (Colins et al., 2004) that are commonly used nowadays in meetings such as algoraves (Collins & McLean, 2014; Collins et al., 2003). In here, we can see an evolution from bands like The Hub with state of the art live-coding. The Hub is a band from 1980’s that used San Francisco bay area as a fertile ground for composition experimenting with microcomputers as a musical instrument automata (Brown, 2002), also 2.1 Literature Review 15 growing from the League of Automatic Composers (Boutwell, 2009). A good reference can be Alexandra Cardenas, currently based in Berlin and also part of the live-coding scene in Mexico City (Alex, 2013). Figure 2.7: A picture of The League of Automatic Composers performing in San Francisco 1981 (McKinney, 2016; Brown, 2002) The live coding performances require flexible tools for real-time musical structuring and the tool itself (the coding environment) must not be a hindrance while performing. Just like a classical instrument, the artist needs to have clear control over the engine. Seeking an agile and quick deployment of the content, the artist needs to use frameworks that allow that. Not only for livecoding, but also for rapid prototyping. There are examples like ChucK11 (Wang & Cook, 2003), TidalCycles12 (McLean & Wiggins, 2010), Sonic Pi13 (Aaron & Blackwell 2013), Impromptu14 (Sorensen, 2005) and Fluxus15 (Magnusson, 2014). These allow the fast generation of audio and/or graphics. Some of these frameworks, tools, and artworks exist thanks to the needs of the artist that created them, which is something that happens a lot in digital systems. And sometimes, when making a review of existing music and audio-visual programs we can observe that some aspects of 11ChucK official website https://chuck.cs.princeton.edu [accessed 27/06/2019] 12TidalCycles official website https://tidalcycles.org/index.php/Welcome [accessed 27/06/2019] 13Sonic Pi official website https://sonic-pi.net/ [accessed 27/06/2019] 14Impromptu official website http://impromptu.moso.com.au [accessed 27/06/2019] 15Fluxus official website https://www.pawfal.org/fluxus/ [accessed 27/06/2019] 16 Contemporary and Historical Practice artistic expression especially on electro-acoustic composition are not addressed by standard music programs (Thiebaut et al., 2008). That might be, or not, a problem. 2.1.6 Instruments using Web Technologies By leveraging the internet as a live software ecosystem, this chapter will examine how such technology can best reach artists, and live up to its potential to fundamentally change creative practice in the field. As a starting point compared to the days of today we have the long distance concerts by telephone. Figure 2.8: Picture found on article Long Distance Telephone Concerts by Scientific American, February 28, 1891, page 130. Performers in New York and audience in Newton Massachusetts. Creating platforms where the newest techniques can be used by artists as part of their day-today job is a point of innovation that happens all over the world. Using browser-based tools is one of them. There are some similarities on the research happening now and the first experiments of musical performance using internet connections. For instance, audio streaming for collaboration was a big topic of research in University of Southern California (1993) (Schooler & Touch, 1993), nowadays with big advances in broadband speed. A couple of years later, in June 13, 2002 the University of McGill and Stanford help the first transcontinental jam session utilizing high quality audio and video which later led the University of Southern California develop their own technologies for what they called Distributed Immersive Performance (DIP) (Chew et al., 2004). 2.1 Literature Review 17 Approaching composition as plural behaviour, we have an important milestone in collective creation systems with the piece of software FMOL. In 1997 the Catalan theater company La Fura dels Baus commissioned the experimental artist Sergi Jordà to develop an online collective music creation system (Barbosa, 2006; Barbosa, 2003). The artist could connect to this project and find a web-based database which allow the user to upload pieces to the server for later modification. This project later included the play F@ust 3.0, inspired on Goethe’s work. Figure 2.9: Courtesy of Alvaro Barbosa. A screenshot of FMOL software. A web-based tree-structured database. c Sergi Jordà Nowadays, we have interfaces embedded in the browsers that allow us to control all kinds of computational tasks through many different API’s in or out of the WWW. Recent developments also allow users to make their own libraries and development in powerful languages like JavaScript (JS) (Flanagan, 2002) and it is important for audio-visual artists that work on browser-based technologies to understand the structure of both the Web Audio API (Favory & Serra 2018; Smus, 2013) for audio development and the WebGL API (Parisi, 2012) for graphics and interactivity. // Just as the Internet offers huge potential as a shared musical performance space (Jin, 2015), it also has other great applications for the arts. That can be found in examples like music production on cloud tools (Morandi, 2017), automatic audio mastering (Najduchowski, 2018) and collaborative code editing (Fiala, Yee-Kind & Grierson, 2016). Which is the example of Soundtrap16, LandR17 and CodePen18. Some of these tools found across the internet may not be done specifically to and for artists, and not at all correlated with artistic performance. But might be interesting to study as a potential use case. 16Soundtrap main website https://www.soundtrap.com/ [accessed 27/06/2019] 17LandR main website www.landr.com/ [accessed 27/06/2019] 18CodePen main website https://codepen.io/ [accessed 27/06/2019] 18 Contemporary and Historical Practice 2.1.7 Intermedia and Digital Interfaces There is a lot of interesting work from artists all over the world trying to merge technologies together, and there is also a great desire to extend the human senses in art. Artists of course also explore that field, and one way of doing that is to merge technologies together and messing around with different feelings. An example of that might be the work of Neil Harbisson (2013), audiovisual artist and president of the Cyborg Foundation19 who implanted a sensor in his brain to perceive colors using their frequency range and transmitting it to his brain (Bryant, 2013), this allows the artist to "hear” colors. This intervention in his body was considered one of the most shocking post-human transformations of all times alongside the extreme performances of Yoko Ono and Marina Abramovi´ c (Jones, 2013). The artist Stelarc, pseudonym of Stelios Arcadiou, might also be worth mentioning as one of the most important artists trying to push the boundaries of human perception and capabilities, known for his transformation of the ear in his arm, performances like the “Third Hand” in his body (Paolo & Kirk, 1995) and for texts like “From Zombie to Cyborg Bodies - Extra Ear, Exoskeleton and Avatars”. These extensions are just like the multimodality of an instrument and it’s propagation, sometimes through different media. It’s a barrier that is designed by each medium broken when necessary, from technological methods to the human body. Using the iconic phrase by McLuhan (1964) "We have already extended our senses and our nerves by the various media" we all understand the intentions of these analogies. Coming back to the audiovisual domain, sensory associations between color and music are found across many cultures as well. One of the earliest is the Pythagorean quest to assign a particular color to each musical note, but in more recent times we have more people involved in synaesthetic experiences like Charles Baudelaire, Wassily Kandinsky, Sergei Eisenstein, and Richard Feynman (Ione & Tyler 2003). We know of Kandinsky’s interest in "colored hearing”, and also Nina Kandinsky, his wife, wrote that he passionately loved colors from early childhood and claimed to distinguish a particular smell and musical sound for each color, a common synesthetic trait (Kandinsky, 1947, p.9). These analogies are also a bridge to the word media art and new media art, whatever that means. The structures that are built take advantage of one media as extension of another. The terms transmedia20 and crossmedia are important on this matter but in regarding this particular example intermedia might be more accurate. We have to realize the importance of the term intermedia, brought to us by the composer and poet Dick Higgins, also part of the group Fluxus (Smith, 1998). The expression says that when two or more discrete media fuse conceptually, they become intermedia, they then difference from mixed media, being inseparable from the essence of the work of art (Longhi, 2002; Higgins, 1984:138). 19Cyborg Foundation main website https://www.cyborgfoundation.com [accessed 27/06/2019] 20 2.1 Literature Review 19 Figure 2.10: Kandinsky’s Yellow-Red-Blue (1925) is from his period of totally non-representational works, but this particular painting seems evocative of the concept of a person experiencing vivid ‘synesthetic’ imagery. Oil on canvas, Musée National d’Art Moderne, Centre Georges Pompidou. Reprinted with permission: Réunion des Musées Nationaux/Art Resource, NY. 2003 Artists Rights Society (ARS), New York/ ADAGP, Paris. Amy Ione and Christopher Tyler on Neurohistory and the Arts. 2.1.8 Contemporary Media Art As quoted in the first chapter, this research is carried out in the context of Braga Media Arts, having a very close relation with media art practice. Created in 2004, the UNESCO Creative Cities Network (UCCN) promotes cooperation with and among cities that have identified creativity as a strategic factor for sustainable urban development. With the growth of this network, UNESCO aims to place creativity and cultural industries at the heart of the cities development plans and to promote an active cooperation between them21. Since its foundation, Braga has been in the avant-garde of its different contemporary scenarios, wether it was under the roman empire or during the sumptuous baroque period. With each challenge, the city has become richer in diversity and more curious about near future.22. 21Citation of the UCCN page on Braga Media Arts Website http://www.bragamediaarts.com/en/uccn/ [accessed 27/06/2019] 22Citation from Braga Media Arts official Website http://www.bragamediaarts.com/en [accessed 27/06/2019] 20 Contemporary and Historical Practice 2.1.8.1 Media Arts Definition There are many definitions of Media Arts. For Braga Media Arts we start with the concept of Media Arts as an integration of new technologies into artistic and creative practice, as well as in social exchange and innovation. This includes disciplines such as video games, digital animation, digital and interactive art, sound art, film and digital video, digital television and the use of technology in the performing arts23. 2.1.8.2 Educational Department Goals New technologies have revolutionized the world of artistic creation, extending it to new artists and, above all, new arts. In some cases democratizing while in others isolating. The Braga Media Arts Educational Service takes this space between creation, Media Arts and the community. Relying on new technologies, it encourages the consumption of art, performance, creation and development of tools and knowledge. The goals should be achieved through a transversal programming that serves the community using media as the basis of development24. As a connection link to other media art cities around the world, Braga Media Arts alongside gnration25 gallery has hosted artists worldwide referenced like Ryoichi Kurokawa26, Lucas Paris27 and Nonotak28. Figure 2.11: Image with both the official logo of Braga Media Arts (left) and the official logo from UNESCO’s UCCN (right). 2.1.8.3 Towards Media Arts Practice Wunschmaschine und welterfindung which alludes to two specific aspects of technology is one example for this matter. It reflects in aspects of technology, more specifically to both its visionary potential as an object onto which to project fantasies and desires, having the power to invent the world (Büscher et al., 1999). Also, in 1997, Felderer (Eine Geschichte der Technikvisionen) states that technology-based visions are to be seen as mnemotechnical structures that allow to think the unimaginable, to project an image to the never-before-seen... “Technology as content 23BMA public definition on Media Art practice. 24Public objectives of BMA Educational Service as a diffuser of artistic activity. 25gnration main website http://www.gnration.pt [accessed 27/06/2019] 26Ryoichi Kurokawa main website http://www.ryoichikurokawa.com [accessed 27/06/2019] 27Lucas Paris main website http://lucasparis.ca [accessed 27/06/2019] 28Nonotak main website https://www.nonotak.com accessed 27/06/2019] 2.2 State of the Art 27 devices organized, reproducing different noises, samples, and synthesized sounds from random locations in the hall. This performance is a solid example of what can be accomplished with a node.js server, instantiating various sound generators and propagating in the vector of connected users using mobile phones as playback machines. This performance was also presented at the Herzen University and Cross Art festival (2015), both in St. Petersburg, Russia. This artwork has clear relationships with Telesymphony by Golan Levin and his mode of performative interaction. The cellphones of the public were also remotely controlled by Andrey Bundin. Figure 2.20: Two pictures of the Concert for Smartphones and Orchestra in Georgia Tech (2016) during Web Audio Conference (WAC). On the left Andrey Bundin on electronics and on the right as violin, Alexey Kotchetkov. As said earlier, the internet is a place where people tend to work on a lot of different ideas collaboratively (sometimes singularly), and we can find incredible examples of instruments development sometimes never done in the academic context that actually can be a reference. That is the example of works like Simulacra34, the Chrome Experiments Lab35, the Multi-player Piano36, and Blockdust37. There are also projects that fit in this section but don’t particularly use the web nor web technnologies. Projects like Tempest (Vigroux & Schmitt, 2012); STM Duality (Sanz & Barandun, 2015) and Imposition (Schwarz & Edisonnoside 2012) (Carvalho, 2018). To use technology as a new mean of artistic expression is very important, evolution for the individual by the individual. (Stelarc, 2011). 34Official repository of the author https://github.com/ChadMcKinney/Yig [accessed 27/06/2019] 35Official website for Chrome Experiments lab https://experiments.withgoogle.com/collection/chrome [accessed 27/06/2019] 36Official website for the Multi-player Piano http://www.multiplayerpiano.com [accessed 27/06/2019] 37Official website for Blockdust https://blokdust.com [accessed 27/06/2019] 28 Contemporary and Historical Practice 2.3 Conclusion On the basis of this extensive text, and with this group of references raised, sometimes different in many aspects, a practical part will be performed using some of the characteristics found across all of them. Structuring relationships on collaborative networked systems and AV interaction, we present Akson, a digital environment as product of this research. We will document and explain the whole process of conceptual and technical development on the next chapter of this document. With an approach following the guidelines of practice-led research (Liamputtong & Rumbold, 2008), we create a strong bond with artistic performance, making experiments during the research period. The outcome of all the references raised in this document as literature and as contemporary practice on the field are what we consider to be the foundations of Akson. Chapter 3 Akson Environment Akson (’æk,son) is the digital environment presented in this research, and this chapter presents it’s conceptual and technical development in detail. The following sections are made in order to demonstrate the entire process of evolution with an intensive explanation of the potentialities, their references and the interface. Figure 3.1: Print Screen of Akson took by the authors. 3.1 Overview To start to explain the whole existence of Akson, it might be a good practice to define one of the biggest conceptual reasons it has, as an environment. Akson is a bridge, a linking current between people, between devices and between audio/graphics. Akson is also called so due to the cerebral organization of the Axons. If we look at the neural architecture of a human brain and 29 30 Akson Environment try to understand the structure that neurons create, we realize that the Axon is the medium that a neuron uses to establish contact with another neuron (Arbib & Aleksander, 1995). The nervous tissue (or neural) is composed of nerve cells (neurons), supported by cells in the glia. Neurons are the cells essential for brain activity. In our brains there are billions of these neurons organized in local circuits, which in turn constitute subcortical regions. Neurons have three important components: a Cell Body, Dendrites, and the Axon. Neurons are interconnected in circuits where there is the equivalent of conductive wires (axon fibers of neurons) and connectors (synapses, the points at which axons contact the dendrites of other neurons) (Damásio, 1994). The way the Akson structure is originally thought, is a bit like brain activity, and not just an analogy to the Axons as connection links. The bundles of axons traverse the brain substance in the white mass, connecting different regions of the cerebral cortex. Some regions are local, between regions of the cortex separated by small millimeters, while other connections bind regions that are far apart, such as, for example, cortical regions from one of the cerebral hemispheres to cortical regions of the other (Purves et al, 2004; Damásio, 1994). This is fundamentally how we should look at Akson in terms of scale and network structure. The neurons exist in the network of the brain, that by itself exists in the network of the body, that exists in the network of the society. Akson also exists as a network, in the network of the Web that by itself interfaces with other networks across the world. It also creates smaller structures with people that are currently in the same interaction model (see section 3.7), between devices and between audio/graphics. Figure 3.2: Courtesy of Publicações Europa-América. Image in O Erro de Descartes by António Damásio. Diagram of a neuron with it’s principal components: Cell Body, Axon and Dendrites. This analogy to the human body and the structures in which it is submerged is undoubtedly one of the most important concepts that defines Akson. It is also a possible way of interpreting it as an interface and/or for other types of intersection between different media, spaces and people. 3.2 Akson and Web Technologies 31 3.2 Akson and Web Technologies Akson technologies rely on the Web and it’s programming interfaces, allowing every device that can connect to the internet to use the system. It has server-side programming to define different modes of interaction (interpersonal and singular), and front-end for Audio and Graphics generation/playback. This way it’s possible to have general control capacity between users (artists or audience) and establish interaction methods inside the network. In the beginning of this research the code started to be allocated in the Heroku1infrastructure, a cloud application platform that has similar properties to the Amazon AWS2cloud computing services. And the version control of the code was done through GitHub3, a common software development platform. These interfaces have cross integration to make it easy to deploy digital systems. On the next figure we have a diagram that shows the work-flow adopted for the code development in this research. Figure 3.3: Starting on an open source code editor Atom (bottom-left) and then uploading it to the cloud through SourceTree (bottom-right) an open-source Git GUI software. When the system is fully updated we have the GitHub interface of the Akson repository with all the code commits that by itself interfaces directly with Heroku pipeline. 1https://www.heroku.com/home [accessed 27/06/2019] 2https://aws.amazon.com [accessed 27/06/2019] 3https://github.com [accessed 27/06/2019] 32 Akson Environment Cloud services that offer Unix machines for people to use, normally interface with several programming languages and/or libraries. In this research, we use node.js4(Teixeira, 2012; Tilkov & Vinoski, 2010) for backend programming. This is a JS runtime built on Chrome’s V8 JS engine. A Google’s open source high-performance JS and WebAssembly engine, written in C++ (Gray, 2009). node.js libraries were managed through npm5(Wittern, Suter & Rajagopalan, 2016), static file serving through express6(Bush & Linden, 2016; Holmes, 2015) and the connections between users were managed through socket.io7(Cadenhead, 2015; Rai, 2013; McCaw, 2011), a library for real-time web applications that enables two-way communication between web clients and servers. With the server handling the connections and the behaviour of client pages, it was then possible to start developing the AV system on top of the Web Audio API (Roberts, Wakefield & Wright, 2013; Wyse & Subramanian, 2013) and the WebGL API (Parisi, 2012; Marion & Jomier, 2012). A better description of the AV engine will be done in the next points of the chapter. There are online resources that support the assessment and understanding of this research. The main code (with the progress placed in time) can be found on https://github.com/luisArandas/akson. Documentation regarding the code can be found on https://akson.readthedocs.io/en/latest/index.html (an open-source and free documentation platform). And sound tracks from the public experiments done with Akson (see chapter 4) can be found on https://luisarandas.bandcamp.com/album/aksonenvironment. Figure 3.4: On the left, screenshot of the offical readthedocs website. On the middle, screenshot of the commits tab in GitHub. On the right a screenshot of Akson’s bandcamp with music from public experiments. 3.3 Audio Engine As previously mentioned (section 3.2), the sound system is built from the Web Audio API. This is a high-level Web API for processing and synthesizing audio in web applications. It involves handling audio operations within an audio context, and has modular routing. It is designed carefully to allow a powerful and versatile way for controlling audio on the Web (Kleimola & 4https://nodejs.org/en/ [accessed 27/06/2019] 5https://www.npmjs.com [accessed 27/06/2019] 6https://expressjs.com [accessed 27/06/2019] 7https://socket.io [accessed 27/06/2019] 3.4 Graphics Engine 33 Larkin, 2015; Smus, 2013). Technically, the Akson sound system is made using native JS (Flanagan, 2002) and the library tone.js (Mann, 2015). In these libraries are constructed the two main divisions in the sound system. The Synthesizer (Lead) and an Auto-Filtered Noise generator (Background). These have individual controls on their methods/functions as abstracted classes of the system. There is also the possibility to define the notes the current Synthesizer is going to play, and detailed explanation on the interface will be made in section 3.6. 3.4 Graphics Engine The graphics system is built from WebGL (section 3.2), an API used to create 3D graphics in a Web browser. Based on OpenGL ES 2.0, WebGL uses the OpenGL shading language, GLSL, and offers the familiarity of the standard OpenGL API (Angel & Shreiner, 2011). Because it runs in the HTML5 Canvas element, WebGL has full integration with all DOM interfaces (Anyuru, 2012). Akson has four technically arranged scenes as objects. They have multiple AV properties unique to each other, allowing interaction with the instrument’s methods without requiring a GUI. As the user uses the capabilities of the interface, they are allocated dynamically in the space of the web page. The graphics system uses native JS and the three.js (Dirksen, 2013) library. A project that creates a lightweight 3D graphics library with a default WebGL renderer. 3.5 Instruments of Akson The Akson is originally designed having four main divisions as a group of instrumental potentials. As cited in section 3.3 there are two main divisions in the sound system (Lead and Background), one called Graphics (to control mainly methods in the various graphic scenes) and another called Post-Prod which has general AV controls in the environment. The various distributed methods belonging to each part can be accessed by the Instruments panel found in section 3.6.3.2. The division of Akson’s experience into these four parts is also explored in the various modes of interaction (see section 3.7). 3.6 Interface To experience Akson, the user does not need to deal with code or understand how the technologies described in this document work in the first place. One can access the current host by typing in a browser www.akson.xyz or download the last standalone version from GitHub tab releases. The cited DNS serves the current server used in Heroku. It is recommended to use Chromiumbased browsers, the main target of this project. Chromium8is an open-source web browser project 8Chromium main website https://www.chromium.org [accessed 27/06/2019] 34 Akson Environment in which Google Chrome bases its source code. As mentioned in section 3.4, Akson is divided into four scenes (fig. 3.5). These allow interaction by the computer mouse and/or touch screen, exploring different sound properties/methods of the system. Sonically, the first and fourth scenes (counting in figure 3.5 from left to right) explore methods relative mainly to Lead, and the second and third explore mainly to Background. Figure 3.5: Screenshots of the four graphical scenes of Akson in their default state. In the following sections the relationship between graphical interaction and sonic feedback will be explored in depth. These relationships are also broken down by a custom GUI that can be found when the user connects by Desktop (section 3.6.1). Also, WebGL is a DOM API, which means that it can be used from any DOM-compatible language: JS, Java, or—if you embed WebKit into an application on a Mac — Objective C (Thiemann, 2005). •Interaction on the First Scene –Here the user is able to play notes on the main Synthesizer (see section 3.6.3.2) by clicking the instantiated geometries. When Akson begins, objects are instantiated and it’s possible to formalize interaction through raytracing9. This allows AV feedback from a custom perspective. The instanciated geometries change color on click (for all users). •Interaction on the Second Scene –The second scene lets you control Background properties in the form of rotation of the contained geometry. This scene creates a custom dodecahedron with vertices scattered across space that by itself controls multiple instances of the Background domain of Akson. A more detailed explanation on the methods can be found in fig. 3.9. •Interaction on the Third Scene –The third scene creates a new instance of geometries that occupy the whole field of the web page. These geometries can be controlled by the XY position of the mouse/touchpad and also control methods of the Background domain. By moving through the web page the user can have a dynamic view of the object. •Interaction on the Fourth Scene 9three.js Raytracing/Raycaster documentation https://threejs.org/docs/api/en/core/Raycaster [accessed 27/06/2019] 3.6 Interface 35 –The fourth scene has a run look similar to the first but with a different aesthetic and controlling different methods of the Lead. Here the master synthesizer is played by playing notes when there is human interaction. The instantiated geometries may be in wireframe mode, and the colors may also be different. These visual aesthetic choices then have repercussions on the type of sound played. Regarding AV relationships. As the behavior of these systems may be tied to different factors, a perspective on interaction is not solely focused on action-reaction patterns, but on the overall variable behavior of the work, in each occurrence and in response to interaction (Ribas, 2013). These relationships are also explored collaboratively among users who share Akson, a more detailed explanation in modes of interaction can be found in the section 3.7. 3.6.1 Desktop Akson is an audio-visual instrument that allows interaction on the browser screen directly with the graphics generated, but also has a more conventional GUI composed of dials, sliders and buttons (section 3.6.3). This GUI is found on computers (both Desktop and Laptop) and can be enabled/disabled from the space bar on the keyboard (keyboard functions in detail can be found in section 3.6.3.5). Warranties are not guaranteed to work on tablets (both iOS and Android) due to the volatility of machine types (in size and specs) but it is always possible to call the virtual keyboard. In the next sections we explore the various types of interfaces created for Akson, how they can be used and what they contain. There is a main bar at the top of the window, there are interfaces in Dialog10 format and Modal11 format. They are built using the Nexus open-source project (Allison, 2011) that has a set of nexusUI JavaScript objects (Allison, Oh & Taylor, 2013) demonstrating many dynamic functions that can be done within the canvas object inheriting from HTML5 DOM. Akson has 64 sliders, 109 buttons, 8 dials and bidirectional socket connections for all of them. 3.6.2 Mobile On mobile phones, Akson exists in a specific way and a bit different from computers. It removes all GUI, allowing only the interaction with the scenes through the touch screens. Whenever the server recognizes an input, it questions what kind of interface it is. If it stays on the type of interface recognized as mobile phone, then Akson will run this way. Devices of this kind also allow you to interact with scenes and play content. There is also an adopted characteristic made to allow a large-scale controlled experience. If there is a device in the 10Dialog definition on Computer Hope https://www.computerhope.com/jargon/d/dialogbo.htm [accessed 27/06/2019] 11Modal definition on Computer Hope https://www.computerhope.com/jargon/m/modal-window.htm [accessed 27/06/2019] 36 Akson Environment Desktop/Laptop category (see section 3.6.1) changing the scene, the mobile phones also change. In this way, it is possible to interact remotely (without GUI and without AV content streaming) with large quantities of mobile phones with potentially large distances and to reproduce content in a coherent way. It also allows us to think of them as resonant machines in space (Blanton, 2019; Henriques, 2015) as a distributed array of speakers. The figure on the left is a photograph of a Huawei P10Lite mobile phone connected to Akson. It is an Android smart phone with a screen of 5.2 inches and a resolution of 1920x1080 pixels. With hardware from 8 Core to 1.9 GHZ and running the EMUI 8.0 system. The figure shows the default state of Akson on mobile devices. 3.6.3 Graphical User Interface Apart from all the automatically generated graphical/sonic content and the potentialities in the interaction of the scenes, the graphic interface will be fully explained in this section. It is divided by sectors and they are illustrated with images, tables and text separated by subsections. 3.6.3.1 Master Dialog The Master Dialog. It has effect on the main out of the browser sound/graphics system. Connected to the AudioDestinationNode12, it provides useful conveniences such as the ability to set the volume and mute/stop the entire application. This dialog has 8 dials, 3 buttons and an oscilloscope. On the figure 3.6 we can see (on the left) a print screen of a Master Dialog and (on the middle and right) the System dialog (opened by the About button). This brings up a window with information about the system where Akson is running, and explains the various models of interaction in the network (section 3.7). The information we can get from the system are functions from the navigator and window JS objects and we can compare network information and AV properties of the machine currently playing. The various interaction models are explained with references to Todd Winkler’s book Composing Interactive Music (1998). On the bottom of the figure we have the various widgets listed and their actions. The dialog windows of Akson are minimizable, closable and are draggable, to give freedom to the user. The 12https://developer.mozilla.org/en-US/docs/Web/API/AudioDestinationNode [accessed 27/06/2019] 3.8 Code, Construction and Implementation 43 groups to negotiate these challenges and structure their collaboration." (Freeman & Van Troyer, 2011) 3.8 Code, Construction and Implementation The whole process of Akson’s technological evolution since the beginning of the research was documented from the outset. All the steps taken (commits and uploads to the servers) are recorded so that everyone can see how it evolved in time. It is possible to create comparisons with the decisions taken and to relate them to the experiments throughout the thesis process. GitHub has features as VCS that allow you to revert files back to the previous state (anytime on the research period). On the main Akson repository website https://github.com/luisArandas/akson you can navigate to the Code tab https://github.com/luisArandas/akson/commits/master and navigate the repository in time or compare the new Akson states. Of course in computer programming some steps are taken just to solve logical problems that arise when we set a goal. These steps were taken in order to achieve a bigger challenge in the overall structure of the system, often building the features that today make Akson. On fig. 3.16 we show a diagram of the Akson repository file tree and the folder communication flow. Starting on the Procfile (top figure), the Unix server is told to execute node server.js, and then the server is started. From here a public folder is instantiated with the index.js file that will read the remaining files and bridge dependencies. In the lower left corner we have the layout of the folders and files seen from the root directory and to the right the various file provisions in each folder (docs,public,external and node modules). With this set, we integrate Akson’s GitHub through Webhooks (Morgan, 2018) to dynamically change the server in the United States. 3.9 Conclusion In this chapter it’s done an extensive journey to the concept, development and techniques used in the proposed environment, Akson. Here we demonstrate it’s features, how they work and how they were done. It presents a finished AV environment and its evolutionary process also related to the public experiments of which it was targeted (chapter 4). There is a detailed description of the interface and the actions it has as a tool, with great emphasis on the references that influence the way of thinking that led Akson to be done. The design of a new tool is a virtualization of many actions (Bardon & Malmborg, 1997). When someone designs a tool, instead of focusing on the action s/he focuses on something much more general. What Akson represents and means is subject of an explanation at the beginning of the chapter. It’s demonstrated that the problem to be solved is greater than simply the development 44 Akson Environment of a collaborative system, of a networked system, and even greater than just building this system. It is a creator of networks. A networked environment within networks, creating networks. As an instrument and as a transmutable environment. The analogy to the organizational patterns in the human brain and the extension of the human gesture in the digital world is mentioned as main concept of its existence. It’s also done a full review of the underlying technologies used in this process. The exploration of opportunities provided by the cloud and how that was what originated the easy resolution of the interaction methods used. All the features of Akson, frameworks and libraries used are also explained. Most of all, this chapter serves to understand the various ways of using Akson, how it works and what it represents. It divides the central components of the system in the various sections and subsections, creating pointers between them. 3.9 Conclusion 45 Figure 3.8: Screenshot of the Synthesizer panel followed by two tables with the various widgets functions and methods. 46 Akson Environment Figure 3.9: Screenshot of the Background panel followed by two tables with the various widgets functions and methods. 3.9 Conclusion 47 Figure 3.10: Screenshot of the Graphics panel followed by one table with the various widgets functions and methods. 48 Akson Environment Figure 3.11: Screenshot of the Post-Prod panel followed by two tables with the various widgets functions and methods. 3.9 Conclusion 49 Figure 3.12: Screenshots of the Allocate dialog (top), the Logs dialog (middle-left), the Save Settings dialog (middle-right) and the Top bar with a panel containing methods of action. 50 Akson Environment Figure 3.13: Screenshot of the modes of interaction modal triggered by the Top bar. Figure 3.14: Keyboard map (QWERTY) with a panel containing methods of action, Keycode Events and the functions found in the code. 3.9 Conclusion 51 Figure 3.15: Diagram of Akson’s descenter model. 52 Akson Environment Figure 3.16: Diagram of Akson’s streamed model. 4.2 Experiment at gnration 59 •Most people used the equipment provided by the station and not their own mobile devices to interact with the system throughout the experience. •Apart from musicians and artists that asked questions about the system, who spent more time interacting were children and young people up to 30 years old. •Overall, while interacting with Akson, people were more focused on the graphics until they discovered the capabilities of the GUI. Even if the system had a dedicated screen to show attendees an image with key controls to interact with the keyboard (see fig. 6.4), help was always given to anyone entering the installation space. Most people also interacted verbally when they came across the setup and some participants also used their devices, the most successful ones were Chromium-based browsers (currently default on Android is Chrome). That helped to see the performance on all kinds of devices (Safari wasn’t always successful and a "No MIDI Support" alert was common). Figure 4.2: Courtesy of gnration’s communication manager. Photograph took on the site of the installation during the morning of open day. 60 Akson Public Experiments 4.3 Experiment at Openfield The third experiment was done on May 6th in Openfield8Creativelab, located in the center of Porto, home of a Media Art Collective. It was performed with equipment kindly provided by the Atelier. 4.3.1 The Performance The third public experiment of Akson as a research context happened as a performance. It was done with the help of the musician Francisca Gonçalves (also member of the Openfield collective) and took place in the laboratory. The nature of this experiment was based on a more conventional laptop performance (see section 2.1.5). The two performers played without any previous rehearsal. The equipment was kindly provided by the Atelier, apart from the performers personal material (laptops and audio devices). The system was composed of one computer to each performer, a sound system and a projector. This experiment took 10 minutes and started at 10:15 in the morning with 6 researchers as public. One computer was a MacBook Pro (late 2013) with a processor of 2.7GHZ Intel Core i5, 13.3-inch (2560x1600), RAM memory of 8GB 1867MHZ DDR3 and a Graphics card Intel Iris Graphics 6100/1536MB running OSX Mojave 10.14. The second computer was of model MacBook Pro (late 2015) with a processor of 2.2GHZ Intel Core i7, RAM memory of 16GB 1600MHZ DDR3 and a Graphics Card Intel Iris Pro 1536MB running OSX Sierra 10.12.6. The first computer was connected to a Scarlett 2i4 (2nd Gen), an audio interface of 4 way outs up to 192KHZ with 24Bit. The second computer was connected to a Roland Quad-Capture, an audio interface with 2x2 analog outs up to 192KHZ at 24-Bit. These two interfaces connected to the main mixer of model Alto L-12, a twelve channel legacy mixer with integrated DSP of 24-bit that connected to 2 Tannoy Reveal 502. The second computer connected by HDMI to a projector EPSON EB-S05 of 1080p (1920x1080) resolution with 3100 ANSI Lumens. Both the equipment configuration and the performance interaction mode were resolved before it happened. It was defined that the two performers would have two distinct functions during their duration, one focusing mainly on the synthesizer (Lead) and another on the filter-noise (Background). In this way they could explore various resolutions of the work with predefined rules. This is the third and last mode of interaction to be integrated into Akson. As explained in chapter 3.7.3, it divides the central control (dashboard and bidirectional connections) into four parts so that it can be divided between performers. The tests were done in the WI-FI network one hour before the performance and were satisfactory enough so we didn’t have to setup our own network for speed improvement. We ran speed tests that demonstrated a latency of <13MS and 8Official website Openfield http://openfield-creativelab.com 4.3 Experiment at Openfield 61 <4MS of jitter. The download speed was of <45.00MBPS and upload speed of <53.00MBPS. We also had the port 5000 opened to serve Akson. 4.3.2 The Participants Participants involved in this performance as a public had all experience with performing arts and were involved with the academy as researchers. They did not know what method of interaction the two performers took, nor was it explained how the performance would take place, they were only told that if they wished they could connect to our system via the Akson official link, projecting a figure on the wall with the address (fig. 6.5). When the participants shared their views regarding the experiment there was a clear agreement on the various topics mentioned. •Participants liked to know more about Akson and how it works as a system, not just staying for the content it created. •They wanted to know exactly what the performers were doing, separately. •They agreed that, given the aesthetics of the sound and the graphic load, the performance should take more than 10 minutes. Compared with the other experiments, the audio-visual relations were more accurate. There was clear graphical feedback when some kind of sound was generated. However, this kind of performance always brings a great abstraction of embodied personal interaction. When the performance was over, impressions were exchanged on how Akson worked, also to match the participants first overall impression. There were no external connections to the two performers 62 Akson Public Experiments Figure 4.3: A photograph of the performance at Openfield. Courtesy of Marcelo Sousa, researcher and public. 4.4 Performance in xCoAx 2019 As the next public validation, at the beginning of July, an already accepted networked performance will take place on xCoAx92019. An international conference on Art and Computation in Milan, Italy. Here, alongside musician Gilberto Bernardes, will present two different interaction models based on the architectures of Akson. We will explore The Conductor Model, the paradigm of the Symphony Orchestra where the conductors are the master controllers shaping the dynamics and directing time flows. And the Free Improvisation method, based on the free jazz movement of the sixties that produced performances that were highly interactive, spontaneous, expressive and unpredictable (Winkler, 1998). Dynamic control will be done in the two modes of interaction that are established with the audience of the venue or with anyone who can join Akson globally. It is presented in the section 5.2 a future development of a midi learning system using WebMIDI so the artists can connect different devices to Akson and map the controls to the GUI. This 9Official xCoAx website http://xcoax.org [accessed 27/06/2019] 4.5 Conclusions 63 can be achieved using public open-source API’s. Akson already accepts MIDI, but is specific to the authors peripherals and there is no way of introducing different ones without hard coding. 4.5 Conclusions In this chapter the public experiments of Akson were presented. As a collaborative digital interface the three modes of interaction (see section 3.7) were explored with it’s process documented. These experiments, above all, served to develop the final version of Akson presented in this document. Much of the feedback obtained was analyzed as reflection material. The great decisions made throughout Akson’s development have always been based on this process as a creative loop (something that can also be seen on the official repository). An example of that are the interaction modes chosen. Not only based on references, they were the result of common opinions regarding interpersonal collaboration after the experiments. The proposed interaction methods were all publicly tested within the time proposed as research (one school year), and are technically finished allowing them to be tested and demonstrated to the world. 64 Akson Public Experiments Chapter 5 Conclusion In this chapter we present the general conclusions of the research. A list of achieved objectives is made and a set of future work is proposed. The chapters presented between the introduction and the present one are the description of the path this research has taken and the underlying technologies used. A work done in the light of the BMA group in AV collaborative web interfaces. We approach the proposed problem and research question in a practical way presenting a finished, open-ended, free and cross-platform solution to the world. The Akson environment. Here, also target of several future work proposals. This research had a great deal of software development which suffered some adversities. Akson is a system with several features in different fields and we tried to match the same level of complexity, aesthetics and functionality in all of them. All these issues were resolved by keeping a record in time and with the help from artistic experiments. The objectives for which we have proposed the research were carried out, and we maintained a very careful relationship with the possible contributions so that the results could be achieved in the proposed time scale. 5.1 Contributions There are several contributions in this research that can generate conclusions. •A web-based collaborative AV environment that can serve as structural fabric in collaborative artistic performance and/or interactive installations. •A public course in software development that can serve as map for future research. Both front-end and back-end. •A collaborative digital space that offers three different modes of interaction between performers/artists/audiences, publicly tested and free to use. •Duly detailed public experiments that can serve as artistic or scientific inspiration/exploration in future projects. 65 66 Conclusion •The performances and the installation done increased interest and attachment to the local artistic community. •Artistic experiments that materialize a custom aesthetic pattern, chosen by the authors. •An AV environment that can extend the artistic gesture through the cloud with potentially large distances across the globe. •An AV generation stream that can be used as content disseminator by large numbers of devices as resonant and visual machines in space. •An online space of free access that can be used by anyone in the world who has access to the internet. The various contributions listed can always have ramifications between them. They can point to various types of future work and in various areas. In the next section we will list several possibilities as future research. 5.2 Future Work After the work introduced, documented and completed come up some options that can be taken as future investigative work. We, as artists and researchers, have come to various conclusions regarding the path that this project can take. Based on reflections taken in relation to the work done, we propose ramifications in various areas. Some more linked to scientific development and others more related to artistic experimentation. We believe that Akson has the potential to be integrated into more domains apart from what this dissertation explores and that can be extended to other areas. Here we will list the various areas that we consider relevant work. •A good exploratory continuation of this research would be the study of more modes of interaction and their integration into Akson. Winkler’s proposed methods (1998) can be one great reference. •A branch of work more focused on the AV relationship and with a more detailed theoretical development in historical events. For example in color organs and synergies between sound and image also found in the work of Rodrigo Carvalho (2013). •A development of work not only exploring the internet and its infrastructure as medium but also as art in itself. The fundamental structure that defines Net Art. A great reference on that can be the book The Art Happens Here: Net Art Anthology (Connor, 2019). •The development of Akson as an automatic content generation system for exploration of large-scale sound sculptures taking advantage of the internet as a medium. 5.2 Future Work 67 •Synthesizer note generation through stochastic modelling such as variable markov models (VMM’s), allowing the ability to create musical phrases between people without deterministic induction (section 3.6.3.2). •The development of a peripheral control learning system using WebMIDI (Wyse & Subramanian, 2013). •The possibility of streaming sound and graphics through WebRTC technology (Johnston & Burnett, 2012). •Explorations in a performative context in which the Akson is used as an instrument in conjunction with other classical instruments (e.g. violin, cello). •The study of distributed electroacoustic reproduction systems with complex arrays of machines (e.g. computer and/or mobile phones). •We underline that one of the great future possibilities of Akson may involve its integration into artistic practice outside the academic and scientific context. •The use of Akson as a system of analysis and composition is also something we consider important. All of these future work possibilities cited here were specifically designed for the Akson environment. These are projects that, based on our reflection after the development of this research, we consider potentially interesting for future work. These may give rise to varied contributions, in different ways, in different areas. 68 Conclusion 6.6 Akson’s Camera Screen shots 75 Figure 6.7: Screen shots of Akson’s graphical system, taken from the virtual camera. These can be found in http://www.luisarandas.org/akson.html. 76 Appendix Chapter 7 Attachment 7.1 Akson signage in gnration Figure 7.1: Photograph of signage arranged at the entrance of the Akson installation in the open day. 77 78 Attachment 7.2 Articles about Akson by gnration Figure 7.2: Article about Akson found on gnration flyer. Figure 7.3: Article about Akson found on gnration website. 7.3 Vertical flyer of open day 79 7.3 Vertical flyer of open day gnration www.gnration.pt Sexto aniversário celebrado com um conjunto de atividades gratuitas para todas as idades. vários locais · 10:00-04:00 gratuito (sujeito à lotação dos espaços) música · instalação · atividades educativas 27 abril 2019 www.studiodobra.com JOÃO PAIS FILIPE ESCOLA DO ROCK GRAHAM DUNNING SENSIBLE SOCCERS QUADRA AMMAR 808 ALGOBABEZ DJ K-SETS WAVELENGHTS OF LIGHT POR MAOTIK TOTAL ECLIPSE POR DIANA POLICARPO MACHINE MESSAGE POR BERRU ESTAÇÃO DE EXPERIMENTAÇÃO: AKSON MINI MAPA SONORO MÚSICA ATIVIDADES EDUCATIVAS INSTALAÇÃO Figure 7.4: Courtesy of Ilídio Marques, gnration Communication manager. A high-resolution vertical poster image that was found across the city of Braga as publicity. 80 Attachment 7.4 Horizontal flyer of open day Figure 7.5: Courtesy of Ilídio Marques, gnration Communication manager. A high-resolution horizontal poster image of the event. Was also found across the city of Braga as publicity. 7.5 Picture of Akson’s Installation 81 7.5 Picture of Akson’s Installation A photograph kindly provided by the gnration organization. It was taken on April 27 and shows part of the exploratory station. Courtesy of Ilídio Marques, gnration communication manager. Figure 7.6: Courtesy of Ilídio Marques, gnration Communication manager. A photograph of users interacting with Akson during the open day. 82 Attachment 7.6 Picture of Openfield performance Figure 7.7: Courtesy of Marcelo Sousa, researcher and public at Openfield experiment. A photograph of the explanation given after the performance, maintaining an interaction with Akson using a mobile phone. 7.7 Article about Akson on Rimas e Batidas 83 7.7 Article about Akson on Rimas e Batidas Figure 7.8: Article about Akson found on https://www.rimasebatidas.pt/gnration-celebra-sextoaniversario-com-open-day-recheado-de-musica-e-actividades/. A post by ReB Team in February 22th. 84 Attachment 7.8 Photograph of gnration’s lineup Figure 7.9: A photograph of a gnration flyer containing the activities of open day. References 91 (Pennycook, 1985) Pennycook, Bruce W. "Computer-music interfaces: a survey." ACM Computing Surveys (CSUR) 17.2 (1985): 267-289. (Puckette, 2007) Puckette, Miller. The theory and technique of electronic music. World Scientific Publishing Company, 2007. (Pulkka, 2005) Pulkka, A. Spatial Culling of Interpersonal Communication within Large-Scale MultiUser Virtual Environments - Master of Science Dissertation. 1995. Human Interface Technology Laboratory - University of Washington. (Purves & Williams, 2004) Purves, D. & Williams, M. (2004). Neuroscience 3rd Edition. Sinauer Associates Inc (July 1, 2004) (Quaranta, 2013) Quaranta, Domenico. Beyond New Media Art. Lulu Press, Inc, 2013. (Rai, 2013) Rai, Rohit. Socket. IO Real-time Web Application Development. Packt Publishing Ltd, 2013. (Ramsay & Paradiso, 2015) Ramsay, David B., and Joseph A. Paradiso. "Grouploop: a collaborative, network-enabled audio feedback instrument." NIME. 2015. (Reilly, 2013) Reilly, Kara. Theatre, performance and analogue technology: historical interfaces and intermedialities. Springer, 2013. (Ribas, 2013) Ribas, Luísa. "Audiovisual Dynamics: An approach to Sound and Image Relations in Digital Interactive Systems." xCoAx : Proceedings of the first conference on Computation, Communication, Aesthetics and X, Bergamo, Italy 2013 (Roberts, Wakefield & Wright, 2013) Roberts, Charles, Graham Wakefield, and Matthew Wright. "The Web Browser As Synthesizer And Interface." NIME. 2013. (Rost et al., 2009) Rost, Randi J., et al. OpenGL shading language. Pearson Education, 2009. (Sá, 2010) Sá, Cristina Fernandes Alves de. "O que é o interface? Da entificação á identificação do interface enquanto complexo mediador." (2010). (Sá, 2016) Sa, Adriana de Almeida Portela Viana. A Perceptual Approach to Audio-Visual Instrument Design, Composition and Performance. Diss. Goldsmiths, University of London, 2016. (Scavone, 2002) Scavone, G. P. (2002). RtAudio: A Cross-Platform C++ Class for Realtime Audio Input/Output. In Proceedings of the International Computer Music Conference, Gotborg, Sweden, pp. 196-199. ICMC. (Scientific American, 1891) Scientific American (1891a). A long distance telephone concert. [accessed 27/06/2019] (Schooler & Touch, 1993) Schooler, Eve M., and J. Touch. "Distributed music: a foray into networked performance." International Network Music Festival, Santa Monica, CA (1993). (Senna & Aroza 2017) Senna, Guillermo, and Juan Nava Aroza. "Daria: A New Framework for Composing, Rehearsing and Performing Mixed Media Music." 92 References (Shannon, 1948) Shannon, Claude Elwood. "A mathematical theory of communication." Bell system technical journal 27.3 (1948): 379-423. (Smith, 1998) Smith, Owen F. Fluxus: The history of an attitude. San Diego: San Diego State University Press, 1998. (Smith, 2009) Smith, Hazel, ed. Practice-led research, research-led practice in the creative arts. Edinburgh University Press, 2009. (Smus, 2013) Smus, Boris. Web Audio API: Advanced Sound for Games and Interactive Apps. " O’Reilly Media, Inc.", 2013. (Sorensen, 2005) Sorensen, Andrew. "Impromptu: An interactive programming environment for composition and performance." Proceedings of the Australasian Computer Music Conference 2009. 2005. (Stallabrass, 2003) Stallabrass, Julian. "The aesthetics of net. art." Qui Parle 14.1 (2003): 49-72. (Stelarc, 1995) Stelarc Interviewed by Paolo Atzori and Kirk Woolford for Academy of Media Arts, Cologne, Germany, 9/6/1995. [accessed 27/06/2019] (Stelarc, 2011) Stelarc. (2011). Art, Design, Future of Humanity - Interviewed by Adam Ford [accessed 27/06/2019] (Stelarc, 2014) Stelarc. (2014). Zombies, Cyborgs & Chimeras: A Talk by Performance Artist, Prof Stelarc - Curtin University [accessed 27/06/2019] (Teixeira, 2012) Teixeira, Pedro. Professional Node. js: Building Javascript based scalable software. John Wiley Sons, 2012. (Tilkov & Vinoski 2010) Tilkov, Stefan, and Steve Vinoski. "Node. js: Using JavaScript to build high-performance network programs." IEEE Internet Computing 14.6 (2010): 80-83. (Thiebaut et al 2008) Thiebaut, Jean-Baptiste, Patrick GT Healey, and Nick Bryan-Kinns. "Drawing Electroacoustic Music." ICMC. 2008. (Thiemann, 2005) Thiemann, Peter. "A type safe DOM api." International Workshop on Database Programming Languages. Springer, Berlin, Heidelberg, 2005. (Wang & Cook 2003) Wang, Ge, and Perry R. Cook. "ChucK: A Concurrent, On-the-Fly, Audio Programming Language." ICMC. 2003. (Winkler, 1998) Winkler, Todd. Composing interactive music: techniques and ideas using Max. MIT press, 2001. (Wittern, Suter & Rajagopalan, 2016) Wittern, Erik, Philippe Suter, and Shriram Rajagopalan. "A look at the dynamics of the JavaScript package ecosystem." 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 2016. (Wyse & Subramanian, 2013) Wyse, Lonce, and Srikumar Subramanian. "The viability of the web browser as a computer music platform." Computer Music Journal 37.4 (2013): 10-23. References 93 (Ziccarelli, 2002) Zicarelli, David. "How I learned to love a program that does nothing." Computer Music Journal 26.4 (2002): 44-51.