Traces.js: A Javascript library for presenting music, physiology, and other time-series on the web

Traces.js: A Javascript library for presenting music, physiology, and other time-series on the web Lawrence Fyfe King’s College London lawrence.j.fyf[email protected] Elaine Chew King’s College London elaine.che[email protected] ABSTRACT This paper describes Traces.js, our custom-built JavaScript library for presenting music and time-series data on the web. Using Traces.js, developers can easily build a variety of applications for taking in time-series data (audio, MIDI, physiological signals, etc) that, once imported, can be heard, seen, and interacted with. We discuss the origin of Traces.js as the code base for CosmoNote, a citizen-science web application for annotating music, feature, and related physiological data, and our need for a more generalized library for developing further web projects with different functionality and intended audiences. After a discussion of related work, we describe the software design concepts of Traces.js, showing, in detail, how its class-based design can be used to import, present, and export data. We then describe three significant projects that we have built with Traces.js, including the specific features used for each project. In the conclusion, we express our hope that the web audio community will find Traces.js useful for building their own projects. 1. INTRODUCTION Traces.js1is a JavaScript library for building web applications that take in existing data (audio, MIDI, physiological signals, etc) that, when imported via Traces.js, can be heard, seen, and interacted with. Traces.js was developed for the COSMOS2project, starting as code for CosmoNote [3], a citizen-science web application for listening to, visualizing, and annotating music, feature, and related physiological data. To further our research for COSMOS, we needed to create web applications beyond CosmoNote. So we refactored CosmoNote’s code and used it to create the Traces.js library. Traces.js is a client-side library written in plain JavaScript and is designed and implemented for building client-side applications. As such, it can be implemented in any browser by any developer who can program in JavaScript. The design goal of Traces.js is to make it as accessible as possible 1traces.isd.kcl.ac.uk 2cosmos.isd.kcl.ac.uk Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: owner/author(s). Web Audio Conference WAC-2025, November 19–21, 2025, Paris, France. © 2025 Copyright held by the owner/author(s). without requiring the installation of any complex software environment other than a web browser (meaning that, for example, Node.js3is not supported). As such, the design of Traces.js places the library in the context of what the World Wide Web Consortium calls the Open Web Platform4. The only dependency for Trances.js is D3 [1], a visualization and interaction library that can be embedded (like Traces.js itself) in a web page via the standard script element. The rest of the paper is organized as follows. In Section 2, we discuss some related work. In Section 3 we discuss the implementation of Traces.js in detail. In Section 4, we provide an overview of the projects that we have built with Traces.js followed by our conclusion in Section 5. 2. RELATED WORK The main inspiration, in terms of features, for Traces.js is Sonic Viualiser [2]. Since CosmoNote, the progenitor to Traces.js, is a citizen science project, we needed to make it widely available without participants needing to install any software. Thus, we chose to make CosmoNote available as a web application. Because we were working exclusively on the web, from the very beginning, we wanted Traces.js to be something like Sonic Visualiser for the web. However, in contrast with Sonic Visualiser, we also wanted a modular library that would enable the creation of a variety of different applications, with each application having some subset of the total number of features. So, while we tried to match many of the features in Sonic Visualiser, Traces.js has a completely different design philosophy and target audience. The WavesJS [5] library has many features in common with Sonic Visualiser but is suitable for developing web applications and is written in a modular way. For rendering visual components, it uses SVG. Another significant library, in terms of features, is Wavesurfer.js5, a web-based tool for visualizing audio. Wavesurfer.js can be extended in its core functionality with plugins. The fav.js [7] library handles the display of audio and feature data. Audio-annotator6is a library for annotating audio files based on Wavesurfer.js. None of these web tools and libraries was able to handle every type of data (especially MIDI data) that we used for our projects out of the box. Wavesurfer.js does allow plugins but those are implemented in Node.js and we wanted our client-side library to work easily in a web browser without recourse to a third-party content delivery network. 3nodejs.org 4w3.org/wiki/Open Web Platform 5wavesurfer.xyz 6github.com/CrowdCurio/audio-annotator 3. IMPLEMENTATION To start using Traces.js, a developer needs to download the library, traces.min.js, and then to embed it in a web page with the script element. D3, available alongside the minified Traces.js library, also needs to be embedded. Then the features of Traces.js are available via a rich set of JavaScript classes. The process for handling data in Traces.js can be broken down into three steps: 1) importing data, 2) presenting data, and 3) exporting data. 3.1 Importing Data Traces.js can import various kinds of data, including audio, MIDI, and signal (or any other time-based) data. In general, data is imported directly from files via the File API7 or the fetch API8. 3.1.1 Audio Trances.js can import any audio file format supported by the Web Audio API’s decodeAudioData() function. All imported audio files are converted to an ArrayBuffer and then fed to decodeAudioData() by the AudioFilePlayer class. Since loading files is promise-based, AudioFilePlayer has an isReady() method. The following pseudo-code shows how to execute code when an audio file has been loaded by the AudioFilePlayer: audioFilePlayer.isReady().then(() => { // code to execute when ready // draw a waveform, for example }); 3.1.2 MIDI Traces.js can import MIDI files via the MIDIFile class which supports the full MIDI file specification[6] and thus enables the loading of any standard file. When loaded, MIDI notes (and a limited selection of control messages) can be displayed as discrete events and MIDI note events can be heard by employing the Synthesizer class or a subclass as described in 3.2.2. 3.1.3 JSON and CSV Traces.js can load virtually any signal or other time-series data and display it visually if the data is provided as a series of time-value pairs. The easiest way to import data is by formatting it as JSON, according to the Traces.js data format, though Traces.js can also import data directly from CSV files. 3.2 Presenting Data Here, we describe the key classes for presenting data imported into Traces.js, namely the Timeframe class that establishes the main timeline and the various pane classes for handling the relevant data types. The relationship between aTimeframe and the panes that it contains is shown in Figure 1. 3.2.1 Timeframe The Timeframe class is the main entry point for developing with Traces.js since it both establishes a main timeline and serves as a container for the panes that handle imported 7developer.mozilla.org/en-US/docs/Web/API/File API 8developer.mozilla.org/en-US/docs/Web/API/Fetch API Figure 1: A Timeframe that contains three panes: an AudioPane, a VisualPane, and an AnnotationPane. data. The main timeline in Timeframe can expand (but not contract) based on the data imported into any of the panes it contains. Any number of panes can be created and added to a Timeframe with the addPane() method. For example: const timeframe = new Timeframe(); const audioPane = new AudioPane(); timeframe.addPane(audioPane); By default, panes are layered on top of each other as implied in Figure 1. Note that the panes are named for their data. For example, the AudioPane class handles audio data, the VisualPane handles visuals, and the AnnotationPane handles annotations. Drawing any class in Traces.js, including Timeframe, requires two steps: 1) initialization and then 2) drawing. The separation between initialize() and draw() methods is to make redraws less resource intensive since a given class will likely be redrawn many times but would only need to be reinitialized when the vertical size of a Timeframe is increased or if a Timeframe is moved from one parent element to another. Initializing a Timeframe requires four parameters: a parent element on a web page (like a div), the desired width of the Timeframe in pixels, the height in pixels, and the end time in seconds for determining the extent of the main timeline axis. For example: timeframe.initialize( parentElement, width, height, endTime ); timeframe.draw(); Note that the Timeframe must be initialized before it can be drawn with a call to its draw() method. After any panes are added, calling the draw() method on the Timeframe will also call draw() for all of the panes that it contains. Anything that those panes contain (visuals, annotations) will, in turn, also be drawn, making it easy to draw or redraw all data with a single call to Timeframe.draw(). Calling certain other methods on the Timeframe will also change its panes. For example, zooming into a time range in a Timeframe will zoom all panes (even audio panes). Resizing a Timeframe vertically will also resize all of its panes vertically. When panes are added to a Timeframe, they are stacked on top of each other by default. As such, because different panes need to use mouse input in different ways, an active pane needs to be selected to receive mouse input using the setActivePane() method. This places the specified pane on top of the others, allowing it to receive mouse input. timeframe.setActivePane(audioPane); The Timeframe can use the Grid visual class in a way that ensures that the Grid is always synchronized with the main timeline and that it is drawn behind any other panes and their visuals. A grid can be initialized with a Timeframe by using the following option: timeframe = new Timeframe({includeGrid: true}) For cases in which data loading is expected to take a significant amount of time, the Timeframe has a loader animation. The loader animation is made up of a random waveform that pulses until the data is loaded. The following line of code will enable the loader animation. Note that this must be done before calling draw(); timeframe.showLoader = true; timeframe.draw(); When, for example, the loading of a file is finished and a promise is returned, the loader animation is switched off and the Timeframe is redrawn. timeframe.showLoader = false; timeframe.draw(); Figure 2 shows a screenshot of the loader. Without the pulsing animation, it looks like a bar-style waveform. Figure 2: A screenshot of the Timeframe loader animation used with the PhysmoNote application. 3.2.2 AudioPane The AudioPane class handles audio and interactions with audio. It requires an AudioContext to do that: const audioContext = new AudioContext(); const audioPane = new AudioPane(audioContext); Each AudioPane has a player associated with it. The following code would set the AudioFilePlayer as the player for the AudioPane: const audioFilePlayer = new AudioFilePlayer( audioContext, audioFile ); audioPane.setPlayer(audioFilePlayer); Once the audio file player is set, the AudioPane has methods for playing pausing, and stopping audio. The AudioPane allows skipping within a file by a click of the mouse at the desired time (with immediate playback at the new audio position if the audio is not paused or a simple of change of position if it is paused). During playback, the AudioPane shows an animated play head (which can be disabled). Another player for the AudioPane, the SynthesizerPlayer, is mainly designed for playing notes from MIDI files. const synthesizerPlayer = new SynthesizerPlayer( audioContext, synthesizer, notes ); audioPane.setPlayer(synthesizerPlayer); The SynthesizerPlayer needs two components (besides an AudioContext): a synthesizer and the notes to play with the synthesizer. Notes are generally derived from a MIDI file. For hearing the notes, Traces.js comes with two synthesizer classes, a simple sine-based synthesizer called Synthesizer and a subclass called WavetableSynthesizer that loads periodic wave data. All of the playback methods offered for audio files (playing, stopping, pausing, and skipping) also work for the SynthesizerPlayer. There is a difference in how playback is handled with the SynthesizerPlayer compared to the AudioFilePlayer. All notes played by the SynthesizerPlayer must be scheduled as ScheduledAudioSourceNode objects. Since having many notes can cause problems with playback, the SynthesizerPlayer uses a look-ahead scheduler [8] via setInterval() to schedule notes to be played in the specified look-ahead interval. The Synthesizer (or any subclass) can also be played outside of the SynthesizerPlayer via any kind of mouse or keyboard event. For example, we created a clickable SVG keyboard that can both play notes via the WavetableSynthesizer and can also be used to change the values of MIDI notes. The BufferPlayer will play an AudioBuffer such as the one returned from a call to decodeAudioData(). As such, the BufferPlayer is used internally by the AudioFilePlayer. For instances in which an AudioBuffer is available from a source other than a file, the BufferPlayer can be used directly. const bufferPlayer = new BufferPlayer( audioContext, buffer ); audioPane.setPlayer(bufferPlayer); The BufferPlayer is also useful for scheduling and playing portions of audio since both the start offset time in seconds and the duration of playback can be specified when calling its play() method. bufferPlayer.play(offsetTime, duration) 3.2.3 VisualPane The VisualPane class imports JSON-formatted data and then displays it with a variety of visuals. For example, the following code will create a Curve visual. visualPane.createCurve(jsonData); For a Curve, the JSON data must be formatted as an array of time-value objects: [{time: 0, value: 100}, ...] Note that other visual classes have slightly different formats, though all data in Traces.js is time-based on the xaxis. The following list contains the visual classes currently available for the VisualPane. •Curve is either a line or an area curve. •Grid is for drawing grid lines. •Instants are for drawing sets of events that each occur at a single moment in time (represented by a vertical line) and that can be without values (that is, time only) or can have a value (on the y axis), or can have a range of values (on the y axis). •Notes is for drawing notes, generally derived from MIDI data. •ProgressIndicator is normally used as an audio play head. •Waveform is derived from audio data. Figure 3 shows different visuals in CosmoNote. Figure 3: A screenshot of CosmoNote, featuring examples of Area,Curve,Notes, and Waveform visuals. Note that the Waveform visual gets it data from the BufferPlayer via its reduceAudioData() method. Notes are fully editable, allowing users to add new notes, delete notes, and change note values, velocities, start times, and end times. Changes to notes can be heard immediately by playback via the Synthesizer (or derived class). When a user is satisfied with their changes, they can save the Notes to a new MIDI file as described in Section 3.3. 3.2.4 AnnotationPane The AnnotationPane is for displaying and interacting with annotations. The annotations in the AnnotationPane are user-created rather than automatically generated. As such, the AnnotationPane is normally layered on top of VisualPanes or AudioPanes that provide the data to be annotated. All annotations can be labeled with names or descriptions. Traces.js has the following annotations. •Boundary is a special kind of marker with special properties for marking change points between parts of a performance with varying significance. •Comment is a special kind of marker specifically for leaving comments at time points. •Marker is for marking time points. •NoteGroup is a group of notes from the Notes class. •Region is for marking time ranges. Figure 4 shows all the annotation types in CosmoNote (other than the Marker type), layered over the visual data to be annotated. Markers are shown as the blue lines in Figure 7. Figure 4: The AnnotationPane of CosmoNote with a boundary, a comment, a region, and a note group, all shown in red. 3.2.5 ZoomPane The ZoomPane is an interaction layer that can be added over other layers to zoom the other panes into a selected time range. Time ranges are selected with the mouse and are indicated visually by a grey rectangle. Selected time ranges can be both moved and adjusted in size. While the ZoomPane can be placed over other panes, for our projects, we normally have a separate Timeframe that contains the ZoomPane and aVisualPane. This separate Timeframe provides a context map that always shows the entire visual (like a Waveform) while the zoomed portion appears in a larger Timeframe as shown in Figure 5. Figure 5: The ZoomPane with a time range selected, below the main viewing pane zoomed in to that time range. 3.3 Exporting Data Traces.js can export certain types of data to files. The following subsections describe the currently supported data exports. 3.3.1 MIDI Notes imported from MIDI data can be edited and the changes can be saved back to a new MIDI file via the MIDIFile class which fully supports the MIDI file specification. However, the Notes class used for editing does not currently support MIDI channel editing, so all exported MIDI notes will be on their original channels. 3.3.2 SVG All visuals in Traces.js are drawn with SVG (courtesy of D3) and all visuals can be saved to SVG files. Saved SVG files show whatever a Timeframe contains and so can be used to take snapshots of visuals in any state (zoomed, for example) and will include any annotations. 3.3.3 Annotations Annotations created with Traces.js can be saved for later analysis as JSON data. That JSON data can be saved into an actual JSON file or it can be saved to a NoSQL database such as CouchDB9(as we did in the CosmoNote project). 4. PROJECTS Traces.js is currently the backbone of three significant COSMOS projects, each one handling different kinds of data and intended for different audiences. One thing that the projects all have in common is that their names end in -Note to show that they are part of the same family of software applications. 4.1 CosmoNote CosmoNote10 is a citizen science platform for annotating music and related feature and physiological data. Participants create an account and are presented with collections of piano performances that they can annotate. Figure 6 shows the landing page of CosmoNote. Most of the performance collections feature both audio and MIDI data recorded with a disklavier recording piano. Figures 3 – 5 shows a performance of Fur Elise from the the CosmoNote Training collection. CosmoNote uses the following Traces.js features: •The AudioPane class for playing audio files, including the ability to schedule small audio files that serve as an indicator sound for boundary annotations. •Different visuals includingCurve (area and line), Notes, and Waveform. •The ZoomPane, shown in its own Timeframe, below the main visuals. •Annotation types including Boundary,Comment,Note Group, and Region. •CosmoNote can save visuals to SVG files and annotations to JSON files. The classes originally developed for CosmoNote formed the basis for the creation of Traces.js and CosmoNote itself has since been refactored to use Traces.js. 9couchdb.apache.org 10cosmonote.isd.kcl.ac.uk Figure 6: A screenshot of the CosmoNote landing page where anyone can create an account to start annotating. 4.2 PhysmoNote PhysmoNote [4] is a web application for viewing physiological signals and time-series data directly from the PhysioNet data repository11. PhysmoNote is the first web application we built after creating Traces.js as a separate library. To use PhymoNote, users select a PhysioNet database with a specific set of signals. Figure 7 shows the PhysmoNote interface with the first record of PhysioNet’s Fantasia database selected. The signal viewer shows the first 10 seconds of two physiological signals: electrocardiogram (ECG) and respiration. PhysmoNote uses the time range selection (zooming) feature of Traces.js to show the first 10 seconds of signals (or the entire signal if it is less than 10 seconds). In PhysmoNote, that same time range selection is used to advance the view by 10 second intervals with forward and back buttons and to animate the viewer, scrolling the signal at the browser’s frame rate. PhysmoNote demonstrates other features from Traces.js: •The loader animation built into the Timeframe class, needed because the amount of data retrieved from PhysioNet can be significant. •The Grid class for placing a grid that always fits the time axis while still allowing for customizations like the appearance of horizontal grid lines, the number of vertical divisions between time ticks, and the line color. •The Curve class has optional properties that can be set so that when a Curve is drawn, the properties of the Curve can be shown on the screen. For the two curves 11physmonote.isd.kcl.ac.uk in Figure 7, the properties associated with each Curve are shown in the lower left corner. •PhysmoNote uses the Marker annotation class (a vertical line with optional label) for both annotations taken directly from PhysioNet and for optional user-created annotations. Figure 7: A screenshot of PhysmoNote with the Fantasia database selected. 4.3 RumiNote RumiNote is a web application specifically designed for the COSMOS team to present musical selections for participants to annotate as part of a larger study on the physiological effects of music. Figure 8 shows an example configuration in which RumiNote presents Bach’s Minuet in D to eight participants. We created RumiNote with a deliberately simplified interface to allow participants to focus solely on the task of annotating audio. RumiNote uses the following Traces.js features: Figure 8: The configuration page of RumiNote for presenting Bach’s Minuet in D to eight participants. •The AudioFilePlayer class loads audio files selected by the researchers running the study and the Waveform visual. •The AudioPane class has customizable control buttons that can be placed anywhere on the screen that already work with the AudioFilePlayer class. •Participants in the study can also use the computer keyboard to control the audio (space to start/pause and escape to stop). •Participants can also use the keyboard to place annotations while the audio is playing. 5. CONCLUSIONS From the deliberate simplicity of RumiNote to the ability to display physiological signals in PhysmoNote to the full-featured interface of CosmoNote, we have demonstrated the range and capabilities of the Traces.js library. Traces.js has been essential for the creation of our various COSMOS projects and we hope that the web audio community finds useful features for their own projects. While Traces.js has many useful features, we are still adding new features and refactoring existing code as needed, increasing the utility of Traces.js for presenting music, physiology, and time-series data on the web. 6. ACKNOWLEDGMENTS This paper is part of the project COSMOS that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant number 788960). 7. REFERENCES [1] M. Bostock, V. Ogievetsky, and J. Heer. D3data-driven documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, 2011. https://doi.org/10.1109/TVCG.2011.185. [2] C. Cannam, C. Landone, and M. Sandler. Sonic Visualiser: An Open Source Application for Viewing, Analysing, and Annotating Music Audio Files. In Proceedings of the ACM Multimedia 2010 International Conference, pages 1467–1468, Firenze, Italy, October 2010. [3] L. Fyfe, D. Bedoya, and E. Chew. Annotation and Analysis of Recorded Piano Performances on the Web. Journal of the Audio Engineering Society, 70(11):962–978, 2022. [4] L. Fyfe, C. Fernandes, B. Moody, T. Pollard, and E. Chew. PhysmoNote: Simple Web Viewing, Annotating, and Time Map Navigating for PhysioNet Databases. In Proceedings of Computing in Cardiology 51, Karlsruhe, Germany, 2024. [5] B. Matuszewski, N. Schnell, and S. Goldszmidt. Interactive audiovisual rendering of recorded audio and related data with the wavesjs building blocks. In Proceedings of the International Web Audio Conference, April 2016. [6] The MIDI Manufacturers Association. Standard MIDI Files 1.0, February 1996. [7] G. Roma, A. Xamb´o, O. Green, and P. A. Tremblay. A javascript library for flexible visualization of audio descriptors. In Proceedings of the International Web Audio Conference, September 2018. [8] C. Wilson. A tale of two clocks. https://web.dev/articles/audio-scheduling, 2013.