Jamming with Planet Earth

Jamming with Planet Earth Tae Hong Park∗ Department of Music Purdue University West Lafayette, IN 47907 [email protected] Abstract This paper introduces the piece "Gamyeon III" (2023) and Citygram, a sound sensor network system for capturing and visualizing sound, with applications in music creation and performance. Built on a Node.js server and browser-based sensor nodes, Citygram enables the deployment of sound sensors using standard web browsers and edge computing. The sensor nodes capture locative audio and perform processing at the source; high-level, low-bandwidth data is then pushed to a central server. This soundscape data is accessed via the Citygram puller, a browser-based interface that, through the Citygram bridge software, broadcasts OSC messages on a local computer. This system is utilized in live performances, including Gamyeon III, where SuperCollider III dynamically drives artificial soundscapes, exploring the transformation of public environmental sound into musical form. 1 Introduction Launched in 2011 Park et al. (2012), Citygram is a real-time sound-mapping system that employs a network of distributed sound-sensing nodes to capture and visualize environmental noise. By crowdsourcing acoustic data through strategically placed microphones, Citygram records sound levels, extracts salient frequencyand time-domain feature vectors, and classifies sound sources. This data is used to generate interactive sound maps that detail the sonic profile of urban soundscapes (Park et al., 2016). Citygram has been utilized to study noise pollution (Park and Yoo, 2018) and its impacts on public health, inform urban planning, and contribute to acoustic ecology research. The project has facilitated conversations on soundscapes, environmental awareness, and has inspired artistic interpretations of urban noise. The project also led to the formation of GetNoisy LLC ( https://getnoisy.io ), a startup focused on high-resolution noise tracking, including a system that uses AI to monitor airplane noise with one-second temporal resolution. In the realm of datadriven art, Citygram has proven valuable as a creative tool for generative audiovisual composition and performance, as explored in the following sections. 2 Soundmapping Our World in 3D Citygram has been developed by embracing the concept of Soundmapping Our World in 3D: dataDriven | community-Driven | art-Driven approach for capturing, analyzing, and visualizing environmental soundscapes. By leveraging edge computing paradigms (Park and Yoo, 2017), real-time data collection, and AI-driven analysis, this method provides a geospatial representation of sound in urban and natural environments. The first D, the data-Driven module, focuses on the technological side of the project, treating it as a Big Data initiative. The second D, the community-Driven module, emphasizes public engagement, crowdsourcing sound data to scale the sensor network, similar to how YouTube content is created by the public. Finally, the third D, the art-Driven module, aims to raise awareness of the research—particularly noise pollution—while fostering creative interpretations ∗ Use footnote for providing further information about author (webpage, alternative address)—not for acknowledging funding agencies. Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th Figure 1: Overview of Citygram system. Figure 2: Citygram bridge piping soundscape data via OSC to supercollider of soundscapes through immersive audiovisual experiences, interactive installations, and performances. This fusion of scientific research, public participation, and artistic expression transforms environmental sound into both a research tool and a medium for creative exploration. 2.1 System Overview The Citygram architecture, shown in Figure 1, illustrates various types of computing devices on the left-hand side that function as sound sensors running on conventional web browsers. These include smartphones, tablets, phablets, notebooks, and desktop computers equipped with microphones, including our custom sticker microphones (Park et al., 2017). Together, these devices form an edgecompute sound sensor network, where automatic sound event detection, classification, and signal processing occur locally. The resulting output - including low-level feature vectors, classification results (used in the GetNoisy system), and other metadata - is streamed to the central server at one-second resolution. Archived data can then be accessed via the Citygram puller, as shown on the right side of Figure 1. On the receiving computer, the high-level data is used to drive a soundscape synthesizer built on the SuperCollider III platform. The Citygram bridge facilitates real-time data transfer from the browser to SuperCollider using Open Sound Control (OSC) message protocols, enabling interactive sonic engagement between the client device and the synthesized soundscape. The feature vector-driven soundscape synthesizer, which generates what we call a sinescape layer, is juxtaposed with a live human performative layer, as demonstrated in the piece Gamyeon III (2022) for cello, piano, electric bass, computer, and soundscape data. This approach invites performers to engage with environmental sound data—captured and streamed in real time to the client computer—as a dynamic musical partner. By establishing a connection between live performance and sensed soundscapes from remote locations, this method enables a fluid interplay between musicians and distant sonic environments—what we call "Jamming with Planet Earth." 3 Proposed Work and Performance: Gamyeon III The proposed piece, Gamyeon III (2022), is composed for cello, piano, electric bass, and computer. The version included with this proposal features an introductory segment called Pi, based on the number π . The Citygram sound sensor network system, the soundscape data puller, and the OSC 2 Figure 3: April 33 performing Gamyeon III bridge are used to drive the sonification engine in SuperCollider III. This sonification layer forms a synthetic soundscape we call “sinescape,” akin to how natural soundscapes often become the background layer in environments where background, middleground, and foreground comprise a dynamic whole. In Gamyeon III, the performed portions typically occupy the middle and foreground layers, effectively masking the “sinescape.” As the performed material quiets, the “sinescape” is unmasked, allowing it to emerge into perceptual prominence. One of the key ideas explored in Gamyeon III is the concept of masking - Gamyeon in Korean means “mask” - a principle that also underpins MP3 codecs. The work was composed during the COVID-19 lockdown while the composer was living in New York City. One day, an unusual silence descended upon the city, and everything came to a halt. With the city unplugged, sounds previously masked by the relentless cacophony of machines began to emerge, becoming vividly audible. Gamyeon III attempts to capture the symphonic cacophony of NYC - often called “the city that never sleeps” - where sirens, automobiles, street music, and machines resonate throughout the city 24/7. At the same time, it explores the negative space of noise pollution: those unmasked sounds that surfaced and came to the fore when the mechanical hum of the city was silenced during the pandemic. The proposed piece will be performed by the newly formed trio (((April 33))), featuring the renowned Maria Ahn and Lucia Ahn of the Ahn Trio, along with composer and electric bassist Tae Hong Park. A video of the performance from summer 2023 in Seoul, South Korea, can be found at the following link: https://youtu.be/7BdBKuoqqiQ . The name (((April 33))) itself was inspired by John Cage’s piece 4’33", famously known as the "silence piece," which premiered as a piano solo with 4 minutes and 33 seconds of silence. In our case, the central themes revolve around the exploration of spaces, places, and their sonic traces, with a particular focus on environmental noise pollution. 4 Summary and note on technical production In this paper, we briefly describe the Citygram sound sensor network system and our proposed work Gamyeon III using its data puller module. The technical requirements for this piece are simple: two microphones for the acoustic piano, one microphone for the cello, and a stereo sound reinforcement system. The electric bass is routed directly into the computer. Additionally, Wi-Fi or Ethernet is needed to stream salient acoustic feature vector data, which drives the artificial soundscape generator in SuperCollider III. References Park, T. H., Miller, B., Shrestha, A., Lee, S., Turner, J., and Marse, A. (2012). Citygram one: Visualizing urban acoustic ecology. In DH, pages 313–317. Park, T. H., Shamoon, C., and Yoo, M.-J. (2016). A step closer towards real-time capture of urban soundscapes and noise pollution. In INTER-NOISE and NOISE-CON Congress and Conference Proceedings, volume 252, pages 914–923. Institute of Noise Control Engineering. Park, T. H. and Yoo, M. (2017). Practicable soundmapping: Javascript enabled edge compute. 3 Park, T. H. and Yoo, M. (2018). Citygram one: Seven years later. In 2018 International Computer Music Conference, ICMC 2018, pages 164–168. International Computer Music Association. Park, T. H., Yoo, M., Dye, C., You, J., Buranintu, V., Rekesh, D., and Leonard, I. (2017). Urban soundmapping at the edge. In INTER-NOISE and NOISE-CON Congress and Conference Proceedings, volume 255, pages 5389–5400. Institute of Noise Control Engineering. 4