27th Annual Mersivity / Water-HCI Symposium December Edition: "Breaking Blue Barriers" with Peter Street Basin Master Plan

35 eaking ue 172 rriers 56 Proceedings of the 27th Annual Mersivity / WaterHCI Symposium, Toronto, Ontario, December 4, 2025 Digital Object Identifier = DOI: 10.5281/zenodo.18042961 https://zenodo.org/records/18042961 Peter Street Basin Master Plan MoBaseTM •Clean •Calm •Quiet Proceedings of the 27th Annual Mersivity / Water-HCI Symposium December Edition “Breaking Blue Barriers” Peter Street Basin Maser Plan Online (www.mersivity.com) DOI (Digital Object Identifier): 10.5281/zenodo.18042961 https://zenodo.org/records/18042961 MoBaseTM •Clean •Calm •Quiet PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 2 CONTENTS Symposium Schedule ........................................................................................... 4 Conference Chair’s Remarks + Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Breaking Blue Barriers to Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Peter Street Basin Master Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 State-of-Float: EEG Signatures of Floating in a Water-Walking Ball . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Mersivity in Action: How Digital Waters is Bringing Humans, Technology, and Nature Closer Together . . . . . . . . . 24 Portable Ayinography-EEG Measurement System Using the MuseCroc Mobile Platform . . . . . . . . . . . . . . . . . . . . . . . . . 26 Muse Flow: A Web-Based Humanistic Intelligence System for Audio Mediated Reality . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Let X∈IReality ................................................................................................ 36 XR/XI (Extended Intelligence/Reality) Light Paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Brain Data Visualization in VR/XR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Heart-Rate-Aware VR Meditation Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Safeguarding humanity with sousveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Psyveillance: Cyborg Psychology of Sousveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Social XR: EEG-Driven Visualization in VR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Early Validation of Humanistically Intelligent Exercise System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 DiaShoe: A Smart Shoe for Diabetic Health Monitoring with Gait and Nanoclimate Sensing . . . . . . . . . . . . . . . . . . . . . 71 MuseLog: Real-Time Multi-User EEG Acquisition and OSC Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Exploring EEG as a Unified Modality for Trust Assessment in Human–Robot Interaction . . . . . . . . . . . . . . . . . . . . . . . . 80 Using Chirplet Transforms For Gravitational Signal Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 ColourSenseXR: Helping the Hearing-Impaired Perceive Sound Through Colour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Elev-AI-T.E.: Wearable Technology to Mitigate Bad Elevator Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 3 SYMPOSIUM SCHEDULE Session A: Audio, Acoustics and Sensing •3:00–3:11 — Intelligent Audio Processing for Guitar Effects •3:12–3:23 — Sonar-based musical effects processor •3:24–3:35 — Infrared and ultrasonic-controlled musical effects processor •3:36–3:47 — Chirplet Transform for musical instrument control in Lua •3:48–3:59 — Guitar tuner based on S.W.I.M. •4:00–4:11 — RF (WiFi) S.W.I.M •4:12–4:23 — Musical instrument effects controller(s) array •4:24–4:35 — Brain-sensing music controller with live emotion sensing Session B: Emerging Work in Wearables and XR, Session 1 This session presents preliminary results from ongoing research projects. •4:35–4:45 — Early Validation of Humanistically Intelligent Exercise System •4:45–4:55 — Real-Time Biofeedback for Exercise 4:55–5:00 PM — Washroom Break 5:00–5:40 — Keynotes •5:00–5:10 — Photogrammetry in Peter Street Basin •5:11–5:20 — fNIRS, EEG, and CO2Monitoring •5:21–5:40 — A. Preyra and J. Archbold Session C: Emerging Work in Wearables and XR, Session 2 This session presents preliminary results from ongoing research projects. •6:15–6:30 — Multimodal Measurement Framework for Attentional Modulation •6:30–6:45 — State-of-Float •6:45–7:00 — Brain Data Visualization in VR/XR •7:00–7:15 — Wearable Technology to Mitigate Bad Elevator Design •7:15–7:30 — DiaShoe •7:30–7:45 — ColourSenseXR •7:45–8:00 — Chirplet-Based Analysis of Gravitational-Wave Chirps Session D: Suprahumachines •8:00–8:10 — Holocap Pi •8:10–8:20 — Navigational Assistant •8:20–8:30 — Muse Flow •8:30–8:40 — EEG XR •8:40–8:50 — Gamification of Planking in Unity •8:50–9:00 — Lidar Pi-based Seeing Aid for the Blind PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 4 CONFERENCE CHAIR’SREMARKS + INTRODUCTION Steve Mann Welcome to the 27th Annual Mersivity / WaterHCI Symposium, advancing technology for humanity and Earth+Water+Air... This year’s theme is “Breaking Blue Barriers”. Blue barriers are policies, protocols, and policing that unfairly discriminate against persons with disabilities. An example is the invisible blue border mapped out in the waters surrounding the City of Toronto, forbidding anyone to “swim or bathe” (i.e. to contact the water) except in certain “designated” areas, usually ropedoff, and supervised by lifeguards. While perhaps well-meaning, such policies prevent access, since none of the designated areas are wheelchair accessible, and even if they were, they are all too far from the downtown core where population density is highest, i.e. too far East or West of the downtown core for most people with mobility limitations to practically access. The only accessible locations for water access are areas that are not “designated” so the overall effect of the policy and policing is to prevent persons with disabilities from having any access. Since access to water is a basic human right, “Breaking Blue Barriers” explores how we can break down systemic barriers to access. Safety and accessibility are basic human rights. Recently (Friday 2025 November 28), in the City of Toronto we have successfully broken through the Blue Barrier. So now, rather than forbidding swimming, we must now turn our attention to ensuring the safety of swimmers, though monitoring of pollution in the Inner Harbour, and providing a safe area to swim and launch recreational (as well as therapeutic) watercraft in downtown Toronto (e.g. perhaps a small motorboat exclusion zone in the downtown Toronto core). Moreover, now that swimming is allowed at HTO Beach, we can also repurpose Peter Street Basin (a place that we often used for swimming after many years of cleanup efforts) as primarily a paddleboarding area. We shall discuss this usecase in “Peter Street Basin Master Plan”, further assisting in breaking Blue Barriers to accessible paddleboarding. GENERALIZING BLUE TECHNOLOGY The principles set forth here can form an allegory for technology in general. Consider, for example, platforms or portals like Zoom, which is often used by governments for access to justice, e.g. through a court-of-law. These platforms and portals function like prisons, to imprison users through proprietary protocols. These prisons function quite unlike earlier technologies like the telephone, which were standards-based (e.g. any brand of telephone could place a call to any other brand of phone), or electricity itself (e.g., any brand of light bulb can connect to any electric company’s electrical power. Voltage for light bulbs was standardized in the late 1800s at 100 volts (DC). In order to overcome the resistance of the wires connecting the light bulb to the source of electricity, a slightly higher voltage of 110 volts was supplied, to ensure that approximately 100 volts reached the light bulb. Even the base of the bulb was standardized as a 1-inch diameter screw base, with a standard thread pitch, so that any brand of light bulb could be screwed into any light socket, which could then be connected to any brand of extension cord, using a standard power plug, which in turn could plug into any electrical socket from any electrical power supply company. Today, the 110 volts has evolved to 120 volts, which is still quite close to the original standard established more than 100 years ago. Although some countries use different voltages such as twice the original voltage (e.g. around 220 to 240 volts), one is free to connect devices from any brand, to electrical power from any provider. Some devices even interoperate over a wide range of voltages, and there are many third-party adapters that connect any brand of equipment to any brand of power provider. Standards benefit consumers, the public, and almost everyone except a small number of people who benefit from market capture and extortionary pricing it affords. Persons with disabilities benefit particularly from standards because these persons can modify devices and systems to suit their unique needs and capabilities, or, if they lack the technical skills to do so, they are free to retain the services of others who can. Wheelchairs and other accessibility devices are usually easy to modify because they are often built from older technologies that have advanced more slowly than automobiles and smartphones for example. Fortunately, this slower advancement usually means that the technology is adaptable to a wide range of modified usages to operate for individuals having very different abilities. If we lived in a fair and just society, technology would advance toward standards and interoperability, but unfortunately, we often live in an unfair and unjust society where technologies advance toward “proprieterrorism”, i.e. the terrible tendency of evolving toward proprietary non-standards. Unlike the light bulbs or extension cords of the past, new products, like Zoom, can only be used with Zoom. Thus, if a government agency like a court of law decides to hold a court session over Zoom, individual participants are also required to use Zoom to connect with it and have their day in court. Unfortunately business and governments don’t seem to favour technologies like Jitsi that are standards-based forms of video conferencing. Jitsi is based on free open standars like WebRTC which uses RTMP and RTSP connecting external devices via Jibri or a SIP gateway. A person with disabilities who requires modifications can either make the modifications themselves, or have someone else with expertise do the modifications on their behalf. Similarly with GNU Linux free open-source solutions. Even if an end user doesn’t know much about GNU Linux, he or she can be assisted by someone who does. In this way the PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 5 technology can be adapted to overcome barriers both real and virtual. When a government or business chooses a proprietary product like Zoom or Skype or Microsoft Teams, they are building a Blue Barrier to persons with disabilities. Persons with disabilities can engage in activities to help in “Breaking the Blue Barrier” which benefits everyone, including those without disabilities because everyone benefits from free open standards. We should therefore see the present terrible landscape of government and corporate barriers as a space of opportunity for change, to improve our world, and advance technology for humanity and Earth. One innovation I am proposing is something I call the Latrop. A Latrop is an inverse portal. Wheras a portal is something an organization presents to the individual, the Latrop does the opposite. The Latrop is how an individual presents himself or herself to organizations. Thus a particular individual with a particular disability or a particular set of abilities presents their content on their own personal server, and invites organizations wishing to connect, to use the Latrop. This shifts the burden from the individual who would otherwise need to learn how to use the portal, to the organization which then must learn how to use the Latrop to connect to the individual. The reasoning here is that the larger organizations like governments and businesses have the resources to apply AI models, etc., to figure out how to login to each individual’s Latrop. The Latrop is one of many new concepts and inventions to be presented here. We present here in this Proceedings, the outcome of our Fall / Autumn activities, culminating in the year-end (calendar year) 2025 December 4-21st Symposium. Let me begin with a very simple-to-understand technological situation, namely water access, as a personal narrative, which can be extrapolated and generalized to many other more advanced technologies. THE NEW “DECONOMY” There is a tendency to prioritize economic development above all else. It is often said that the “first world” is preferable to the “second world” or “third world” of economic development, where large buildings replace smaller ones, where large highways replace walking paths, and where beaches are replaced with steel walls to facilitate motorboats. Swimming, cycling, and walking are dangerous to this economic development, because swimmers, cyclists, and joggers are spending less money on cars and motorboats. Cars and yachts cost more than bicycles and swimsuits. Moreover, when people swim and bike and run, they stay healthy. This is dangerous to the pharmaceutical industry, because they need less drugs, less medication, less medical treatment, and spend less time in hospitals. Likewise, the use of GNU Linux is dangerous to the economy because the free open source software world involves less sales of software, software licenses, and subscription-based imprisonment strategies. Terms like “developer”, used to mean a liquid to pour on photographic film to develop a latent photographic image. Now the world more likely connects with “SDK” (System Developer Kit), or as I like to say, “System Detention Kit”, because when we make a distinction between developer and user, we begin to erode the free open flux of science and scientific curiosity. When we venture toward ideas like explainable AI, we again need to be reminded of a simple question: Explainable to whom?. HI (Humanistic Intelligence) should be the end goal, to create systems that are auditable by any end user, not just explainable to some. We present a series of papers that touch on these themes, such as surveillance (centralized oversight) versus sousveillance (distributed “undersight”), and equiveillance (balance between surveillance and sousveillance). This takes us to something I call “Psyveillance” which is a “psyborg” (cyborg psychology) of surveillance, sousveillance, equiveillance, and really all the “veillances”. BREAKING BLUE As economic development runs rampant, maximizing profits of psychological prisons of surveillance, we have AI data centres sucking up our electrical power, and bringing about negative impact on our environment. In particular, the massive energy and water consumption creates a “Blue Barrier” to individuals wishing to live a simple healthy lifestyle. In much the same way that Zoom holds us prisoner, AI in general can hold us as prisoners inside a “Blue Border” of water scarcity, and lack of water access. The best way to love, protect, and celebrate our drinking water is to swim in it, so we begin by “Breaking Blue” (a term one could take as a kind of double-entendre, meaning to run toward blue, as in “make a break for it”, and also to “break through” the barrier). Water takes us into a certain mindset that we can sense using the latest brain-sensing technology. As a co-founder of InteraXon, makers of the Muse, Muse-S, and Muse-S Athena brain-computer-interface (BCI), I like to speak of something that I call “State-of-Float™or Float-State™, which is the mental state one is in while floating on, in, or under water, e.g. in boat or ball, on a board, or in or under the water directly. More generally, I envision what I call an ABC = Airplane, Boat, and Car, a device that can operate on land, water, air, etc., for which I might add something I call “State-of-Flight” or “Flight-State” as another mental state that we can sense using brain-sensing technologies. We present our work on the sensing of various mental states, and brain-sensing technologies as well. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 6 Breaking Blue Barriers to Accessibility Steve Mann Dept. Electrical and Computer Eng. University of Toronto Toronto, Canada Fig. 1: Safety and water access are basic human rights but there’s an invisible “Blue Barrier” blocking people with disabilities from accessing water. Biochemistry is one barrier, for which regular E. coli testing can provide important indicators of safety. Abstract—Safety and water access are basic human rights but there’s an invisible “Blue Barrier” blocking people with disabilities from accessing water. In this paper, we report having finally broken through this Blue Barrier, and outline successful strategies for maintaining water access more generally. THE FALL OF THE BLUE WALL,AND A BIG THANK-YOU TO CITY OF TORONTO! Today marked an important turning point toward the fall of the invisible blue wall that keept many of us out of Lake Ontario. Ontario, a province named after Lake Ontario, is home to the world’s largest (by surface area) freshwater lake, Lake Superior. Superior holds 1/10th the world’s supply of freshwater. I live in Toronto, the largest city on the Great Lakes which hold about 1/5th (21% to be more exact) of the world’s freshwater supply. I’m part of a community of thousands of people in downtown Toronto who swim nearly every day in Lake Ontario. PortsToronto, a government business enterprise, with directors appointed by all three levels of government (Canada, Ontario, and Toronto) defined an invisible blue line in the water that can legally only be pierced with the armour of a vessel, but not with bare hands or feet. It extends one mile due south from the mouth of the Humber River, continuing east, one mile due south of Gibraltar Point on the Toronto Island, and Fig. 2: Within the invisible “blue border” shown on this map, PortsToronto has been prohibiting swimming anywhere that has not been specifically designated as a swimming area by the City of Toronto. Officially-designated swimming areas have been limited to official supervised city beaches. None of these allowed swim spots are wheelchair-accessible, but plenty of forbidden swim spots are accessible. then to the east to a point one mile due south from the eastern boundary of the old City of Toronto, near Victoria Park: For many of us with disabilities or mobility limitations (e.g. inability to cross over sand or mud) this restriction prevents us from swimming. There are also no supervised city beaches in downtown Toronto. Moreover, lifeguard supervision is only available during a limited season and during limited hours. Many of us do cold plunges, cold water swimming, etc., not PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 7 just for recreation, but also due to a medical need, as this is a form of treatment and intervention. Safety is of course important, but however well-intentioned, the Blue Wall is, in some ways, actually dangerous, as it has a chilling effect on healthy exercise and a healthy lifestyle. PIERCING THE THIN BLUE WALL WITH YOUR BARE HANDS AND FEET: The cleanest and safest place to swim in all of Toronto was Ontario Place, home to downtown Toronto’s only pebble beach where thousands of us swam daily until its closure for privatization. In water quality tests, it often outperformed the official beaches in cleanliness. Unlike any of the official beaches, the beach at Ontario Place was also accessible to people in wheelchairs, rollator walkers, and other mobility devices. While there have been claims that official City beaches like Woodbine and Wards have wheelchair ramps, these ramps only go to within about 30m (about 100 feet) of the water’s edge, and therefore only allow viewing the water, but not actual water access. At Ontario Place the security guards would call the Police whenever we swam there. After a while the Police stopped coming, so they began to call the Marine Unit, and a police boat would come and we all got out of the water and waited for it to go away, and then we got back in the water. After a while the Marine Police stopped responding, so the security guards called the Fire Department. Two fire boats came up to the beach, and we all asked them if they would like to join us for a swim. They told us they had received reports that there were people in the water. Eventually they stopped bothering us. We’d finally punched a hole in the Blue Wall. But then the Government of Ontario decided to privatize Ontario Place and chop down more than 1500 mature trees, clearcutting downtown Toronto’s only beachside forest. The trees were cut under cover of darkness, On the evening of Wednesday, October 2, 2024, hundreds of mature trees (around 865) were rapidly cut down at Ontario Place’s West Island (a public park) under the cover of darkness by Infrastructure Ontario, preparing the formerly public space for a large private spa development by Therme Group, sparking outrage from residents and environmentalists who decried the destruction of a unique urban ecosystem and wildlife habitat. All that remains of downtown Toronto’s only beachside forest is a barren island, ripe for cash flow. So many of us started swimming at what the City of Toronto calls “HTO Beach” or “HtO Urban Beach”. Presently it is the last remaining accessible beach in the City of Toronto. It is the only beach where a wheelchair, or other mobility aid can get right to the water’s edge. SAFETY FIRST: We have been testing the water there regularly, and it has consistently been reading as safe for swimming. This week the E. coli counts per 100mL are (West-to-East): 45.0, 21.2, 35.8, 34.0, Fig. 3: Variously sized boats ranging from a 6-foot paddleboard on the left, to a 3-inch “neckboat” (necklace-worn boat) on the right. at four spots spaced uniformly along the length of the water’s edge. These are well within the limit of 100 per 100mL. The geometric mean is about 32.8 which is well under 100. The biggest danger is getting a ticket (being fined), as has happened to some of my colleagues, including the owner and operator of one of Toronto’s best bicycle shops. Many of us wear a surfboard tethered to our waist or ankle, for protection from the Police, since, legally, surfing, or Polynesian Indigenous paddling (laying on a board and paddling with bare hands) is considered boating, not swimming. HOW BIG OF A WIENER DO YOU NEED? Metaphorically, this is like carrying a marshmallow or a wiener (Vienna sausage) so that you can have a campfire, since campfires are only allowed if you’re cooking something. An interesting question becomes “How big of a wiener or marshmallow do you need?” in order to legally have a fire. Here is our collection of vessels, as once mandatory cyborg prostheses, required to pierce the thin blue wall. We once asked the question ”How much boat to I need to wear in order to be allowed in the water.”. TODAY WE NO LONGER NEED TO ASK THAT QUESTION. After many years of correspondence with City of Toronto, from the Mayor Olivia Chow (a paddler herself, as well as a strong supporter of water access rights), to Deputy Mayor Ausma Malik, to Director of Waterfront Revitalization Tom Davidson, and finally, Manager, Community Recreation, Sandra Mccallum, we finally have the definitive answer: PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 8 Fig. 4: “Swoating” (Swimming or Boating) in downtown Toronto. Although these tiny 3-inch necklace boats functioned as little more than “good-luck charms”, they could perhaps be seen as offering protection from fines or citations in Policeinfested waters. As of Today, they are hopefully no longer necessary, because “Breaking Blue” (making a break for the blue lake) is no longer “Breaking Bad” in the eyes of the law. Nov 28, 2025, 6:01 PM from: Sandra Mccallum to: Steve Mann cc: Shawn Page Tom Davidson date: Nov 28, 2025, 6:01 PM subject: RE: Re: Re: Re: Re: Re: Re: Re: Island Accessibility questions Hi Steve, As promised, I spent some time researching this week,and as you mentioned, HTO Park appears to be the most accessible spot in the downtown core for both cold plunges and watercraft launches. I’ll be connecting with our Waterfront Manager to explore ways we can make this location even more user-friendly. Additionally, now that the City is not standing in my way, I contacted Mike Riehl, Toronto’s Harbour Master, asking for advice on the safest way to do my medically-necessary cold plunge activity and he asked two questions: (1) How are you going to get in and out of the water; and (2) What happens if you experience hypothermia. We discussed this for a while, Fig. 5: Now there’s a floating sauna at HTO Beach in the heart of downtown Toronto, right in front of the Rogers Skydome and CN Tower! Hopefully this sauna boat will maintain a hole in the “Blue Barrier” as more people continue to do their cold plunges at HTO Beach which the City and Harbourmaster have finally permitted! and established that the existing safety ladders are suitable for getting in and out of the water safely, and that one should never do cold plunges alone, i.e. always with a group, so that we can look out for each others’ safety. Thus we have finally broken down the blue barrier, and acknowledged that access to water is a basic human right. KUDOS TO THE CITY OF TORONTO: Kudos to the City of Toronto for this burst of commonsense, and not only creating an accessible opening in the Blue Wall, but also exploring ways to make this location even more user-friendly. FLOATING SAUNA AND COLD PLUNGE AT HTO PARK / HTO BEACH: This great news could not have come at a better time, as next month Toronto will soon be home to a floating sauna. I’ve been experimenting with small-scale floating saunas at HTO Beach in the past, unofficially, for personal use among friends, but now there is a large-scale sauna coming soon to HTO Beach. See Fig 5, as well as their website at https://www.loylyfloatingsauna.ca/ Thus the “Blue Barrier” as an invisible blue border wall drawn on maps for years (long before the Leslie Street Spit formed to penetrate the boundary), has finally fallen like the Berlin Wall. Now we must take steps to ensure safe water access, rather than forbid it altogether. Now is a perfect time for Toronto to become a signatory to “Swimmable Cities”. Join us Dec. 4th, as we discuss this huge win for City of Toronto, at our December Symposium, 3pm to 8pm, at the University of Toronto. See links and map in http://mersivity.com Additional Links: Mersivity.com HTObeach.com PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 9 Fig. 5: Prof. Steve Mann with his Headome™invention. The Headome provides underwater eyesight that perfectly matches the refractive index above and below the water, without requiring any special contact lenses as were required with previous similar technologies invented by others. Fig. 6: “BallDome”: using a Mersivity Ball as a sperical observation dome for perfect underwater vision while staying completely dry. completely dry, and provides perfect optical vision, the XR headset can easily facilitate a shared vision. See Fig 7. Photogrammetry and holomargologrammetry are important parts of our research effort, as we build multidimensional datasets for being visualized in a VR (Virtual Reality) or XR (eXtended Reality) headset as well as social XR shared spaces. See Fig 8. Finally, we developed structured lighting technologies to scan the Basin using Lightspace™, which is the tensor outerproduct of a lightfield with a time-reversed lightfield (see Fig. 7: XR-Ball: Perfect shared-vision using XR headsets together with photogrammetry of the Basin. “Intelligent Image Processing”, by S. Mann, published by John Wiley and Sons, in the Interscience Series). Lightspace can be combined with Mersivity Balls, as shown in Fig 9. This method tends to work best in dark conditions, when we have more control over the light sources of our scanning system. B. Underwater acoustic phase-coherent pollution sensing Finally, we developed hydraulophonic pollution sensing which uses underwater sound-wave propagation in the water combined with an advanced transform-based machine learning called ACT (Adaptive Chirplet Transform) to characterize water quality. A nice feature of underwater acoustics is that it is almost completely silent in the air. It is our goal to be able to establish real-time pollution monitoring in the Basin by way of phase-coherent detection of sound wave propagation (infrasonic, sonic, and ultrasonic) combined with optical sensing in the infrared, visible, and ultraviolet spectra. When combined with HDR (High Dynamic Range) sensing, we achive XR (eXtended Reality) providing superhumachine intelligence in the form of XI (eXtended Intelligence) to characterize water quality. See Fig 10 C. Noise pollution reduction Friends of Peter Street Basin is committed to eliminating noise pollution. Prof. Steve Mann is an acoustics professor teaching ECE446 = Audio, Acoustics, and Sensing at the University of Toronto. In keeping with the giving spirit, his acoustics textbook, http://wearcam.org/ece446textbook/ece446textbook.pdf is free and open. It is free for download, and anyone can even look at the source code and contribute using the free open-source LaTeX typesetting language; see http://wearcam.org/ece446textbook/ Paddleboarding in the Basin is wonderful because of the calm water, but there is quite a bit of noise from the streetcar and traffic on Queens Quay that runs right beside the Basin. Therefore, we propose putting vegetation in the derelict abandoned garden areas, to help reduce traffic noise. Ideally we would plant trees there to block sound from the traffic PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 16 Fig. 8: Photogrammetry and holomargologrammetry of Peter Street Basin. (Top) topview; (Middle) Perspective; (Bottom) South entrance to Peter Street Basin. Fig. 9: Mersivity Balls combined with Lightspace™provide perfect optical vision under structured lighting conditions, giving us new technologies to scan not just the shape of the Basin, but how it responds to light underwater. This can provide valuable insight into water pollution. Fig. 10: Hydraulophone (underwater musical instrument at the front of the vessel) can be used for realtime pollution sensing while cleaning up the Basin. Ryan Janzen, the world’s leading hydraulist, operates the hydraulophone while three other volunteers collect garbage from areas of the Basin where pollution is the highest. on Queens Quay. This would dampen sound to mitigate the acoustic resonance affect of the Basin. We are presently studying what trees would best block traffic noise from reaching the Basin, as well as from reaching the adjacent apartment buildings above the Basin. In order to benefit paddlers in the Basin, as well as apartment building dwellers, we are considering tall sound-absorbing trees with fractal leaf patterns that are highly anechoic to dampen resonances that occur at natural frequencies set forth by the parallel edges of the buildings combined with the sound-reflective nature of the PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 17 water in the Basin. Interestingly, the main artistic feature of the Basin was once a sound sculpture, now abandoned. Paddleboarding, in particular, is a quiet, contemplative, and peaceful activity best done in meditative silence. Thus it will help paddleboarders if there is silence in the Basin. The sound-muffling features we propose will benefit not only our paddleboarding community, but also the thousands of residents of the three apartment buildings that overlook Peter Street Basin. This will be a great opportunity for research and teaching on acoustic baffling and generation of a cone-of-silence effect to mitigate streetcar traffic sounds from street-level, at no cost to the City or the region, as it would be research-funded as a research and teaching example for University of Toronto’s ECE446 (Audio, Acoustics, and Sensing). IV. RESULTS After 1,347 days of our CleanupCrew operations, we were all surpised to find that we’d achieved an astounding level of cleanliness. Whereas our goal was simply to mitigate the stench and visual detritus, we actually got the water quality to the level that meets swimmable water standards. Toronto has among the world’s most strict standard which requires E. coli counts below 100 counts per 100mL, whereas outside Toronto the standard is merely 200 counts per 100mL. We were able to get the Basin E. coli counts below 10 counts per 100mL, and typically ranging from 7 to 10. This is about 12 times cleaner than it needs to be for safe swimming. Prior to our cleanup efforts, the E. coli counts were around 30,000 or too high to measure. Thus we made the Basin more than 3,500 times cleaner than it was, in terms of E. coli counts. In other regards such as turbidity, and other indicators, our cleanup efforts netted similar improvements. In order to get highly accurate E. coli counts, Prof. S. Mann worked together with Prof. R. Hoffmann (like Mann, also at the University of Toronto, in the Faculty of Applied Science and Engineering). They took a sample of 500mL and sucked it through a fine filter using an Erlenmeyer flask having a side-discharge. The side-discharge was connected to a vacuum pump to pull all the water through, and then growth medium was applied to the filter. It was then placed in an incubator for 24 hours, after which the count was determined and divided by 5. This resulted in a sub-count precision (an extra decimal place). Moreover the sampling was done at multiple locations in the Basin. To see our testing process in action, here is a video link: https://www.youtube.com/watch?v=vAPSFWjE2Pw See also Fig 11 for a closeup shot of the actual test filter on the top of the flask. We also confirmed our results using various other test methods. As a result, the Basin can now enable safe standup paddleboarding, since, especially for beginners, it is common to fall into the water from a paddleboard. We posted our results on the SwimOP (Swim at Ontario Place) Facebook page which is still active despite the cloFig. 11: Closeup of E. coli test setup for Peter Street Basin. Fig. 12: CBC coverage of our cleanup efforts. sure of Ontario Place. Shortly afterwards, a reporter from CBC (Canadian Broadcasting Corporation), Haydn Watters, contacted us to request an interview. This resulted in a radio piece, a television piece, and a written piece (Fig 12). Shortly after that, we were contacted by BBC who also did a piece (Fig 13), and then by Cottage Life who wrote an article on this work (Fig 14). V. HIGHEST AND BEST USE Whereas the Basin is perhaps not the ideal place for swimming, we believe the highest and best use for Peter Street Basin is as a paddleboarding location for everyone, including those with disabilities. What is unique about the Basin is that the water is calm, with almost no wave action, and almost no currents. Moreover, it is sheltered from the wind. Paddleboarding requires the ability to balance on a board, and this skill can best be learned in a calm protected space like Peter Street Basin. There is no other place like MoBase! Whereas there used to be a pond at Harbourfront (the Natrel Pond) where paddleboats could be rented, it has now been filled in. Love Park (the hart-shaped fountain) is another possible location, but the water quality is poor, and it is too shallow for safe paddling (e.g. risk of falling into shallow water). Much to the City of Toronto’s dismay, Love Park has become a frequent location PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 18 Fig. 13: BBC coverage Fig. 14: Cottage Life for recreational watercraft, especially among persons with disabilities, simply because there is no way for them to access the lake. In this way, the City is in violation of Human Rights Code, R.S.O. 1990, c. H.19. See “Someone brought an actual raft to Toronto’s new Love Park pond” by Jack Landau, in blogTO, 2023jul31. Here are some relevant excerpts: Jaclyn Carlisle, Media Relations and Issues Management Advisor for the City of Toronto, confirmed to blogTO that such activity is a no-no, saying that “While we welcome Torontonians showing Love Park some love, pleasure and paddle craft are not permitted within the pond.” Carlisle reminds hopeful boaters of the painfully obvious fact that a freakin’ Great Lake exists just metres south of the park, saying that, “With more than 45km of Lake Ontario shoreline and a nearby archipelago making up Toronto Island Park, we ask that the public make use of the ample opportunities the City has to offer for urban paddling.” The truth is that there is no other place for persons with disabilities to access the water, other than the City’s fountains. Some of these uses of the City’s fountains are forms of protest or civil disobedience, whereas some of the uses are merely practical acts of desperation because there are no other options. One way that the City could bring itself into compliance with Human Rights laws would be to embrace MoBase, so as to provide a place for persons with disabilities to access Lake Ontario. Peter Street Basin is ideal for paddling, especially paddleboarding. VI. BUSINESS CASE FOR “MOBASE” = MOBILITY BASIN We have already been actively engaged in the cleanup of Peter street basin with the help of local volunteers. As a Professor at UofT, prof. Mann and his students have been actively studying the basin, including water quality etc., with help from Richard McCracken at the City of Toronto who has been a long-time supporter of this project. To further these efforts to clean the basin and provide a long term strategy to keep it clean, we are preparing to form a non-profit and wish to generate funding for that non-profit by renting out accessible paddleboards in the safety of the basin, which is a place where there are no motorboats or waves, and there is shelter from strong currents or winds. We have the support of the local residents, the private buildings adjacent to the basin, local businesses, and Oliver Hierlihy of the Waterfront BIA (Business Improvement Area), which we will show in documentation as a part of our coming application. We are equally preparing an operations and safety plan, including associated insurance requirements. It is our hope that this initiative will both enable/fund the cleaning of the basin, including steps to eliminate the bad smell, and keep the water clean and healthy, but will also help support all the ground level businesses we hope move PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 19 into the surrounding vacant commercial space. People buying drinks at the convenience store or food at small family owned restaurants etc. will all benefit from a Basin that is and smells fresh and clean. VII. RESTORATION ACTIVITIES We have been studying the history of the Basin and are in the process of writing a book about it. Additionally, Friends of Peter Street Basin is in touch with Molson regarding restoration of the Molson Aqua Sculpture. Moreover we wish to restore the aeration system/fountain in order to maintain good water quality in the Basin, and install a variety of sensors and sensing systems for research, teaching, and service to the community. We envison a multi-sensory intelligent fountain that maintains the water quality, along with a prosperous research agenda that puts Toronto first in the world at freshwater research. VIII. TEACHBEACH AND OUTREACH Additionally, we wish to use the Basin as a base of operations for other related non-profit activities such as research on water quality, as well as other related research efforts. For example, we wish to conduct large-scale research on something we call “State-of-Float”, the mind state while floating. We observe that floating induces a measurable neural and experiential state. Studying State-of-Float will help us understand the human condition while paddleboarding. IX. CONCLUSION We described our proposal, MoBase, for an accessible place for paddleboarding, while keeping the Basin clean with a strategy to mitigate or eliminate pollution (water pollution, noise pollution, etc.) to make and keep it calm, quiet, and clean. The highest and best use of Peter Street Basin is accessible paddleboarding, combined with research, teaching, and outreach, setting an example of freshwater stewardship that the rest of Ontario, Canada, and the world will hopefully follow. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 20 State-of-Float: EEG Signatures of Floating in a Water-Walking Ball 1st Steve Mann Elect. and Comp. Eng. University of Toronto 2nd Alexander Vicol Elect. and Comp. Eng University of Toronto 3rd Nagham Sabbour Elect. and Comp. Eng University of Toronto 4th Gesi Wodu Elect. and Comp. Eng University of Toronto 5th Kinkini Monaragala Elect. and Comp. Eng University of Toronto 6th Xiaoming Chen Mech. and Ind. Eng. University of Toronto 7th Patryk Aniolowski Mech. and Ind. Eng. University of Toronto Abstract—The concept “State-of-Float” is used to discuss and measure the experiences of individuals who are floating in different mediums. To study this, participants laid supine inside a transparent inflatable ball while wearing a Muse S Athena headband to collect EEG signals and a virtual reality (VR) headset to standardize visual input. EEG recordings were collected and compared on land (control) and on water, under eyes-open and eyes-closed conditions. Floating was associated with consistently higher theta-to-beta ratios (TBR), an established index of attentional control and arousal, and participants reported feeling calmer. Environmental safety was verified through in-ball CO2monitoring, confirming exposure remained within accepted limits. These findings indicate that floating induces a measurable neural and experiential state: State-of-Float. I. INTRODUCTION Floating environments alter proprioceptive, vestibular, and sensory input, often producing relaxation and changes in attention. In this work, we formalize this experience as Stateof-Float and examine it using wearable neurotechnology in a natural aquatic setting. Participants laid inside a sealed, transparent, 1 m-radius inflatable ball, a consumer device typically used for entertainment and more recently research. This study also connects to ongoing Water-HCI and Mersivity research directions on aquatic embodied interaction and wearable sensing [1]. A VR headset was used to control visual input while preserving ecological validity. This approach builds on humanistic and wearable computing paradigms that emphasize measurement in real-world environments rather than laboratory isolation [1]– [3]. (a) On land (b) On water Fig. 1: Participant, in a supine position, in the ball on land (a) and on water (b). We focus on the θ/β ratio (TBR), which has been linked to attentional control, arousal regulation, and stress modulation [4]–[7]. II. METHODS A. Participants and Apparatus Participants wore a Muse S Athena EEG headband [8] (channels AF7, AF8, TP9, TP10) and a Meta Quest 3 VR headset. Inside the headset, participants viewed a dark scene with a fixed central point “+” to minimize visual variability. Participants laid supine inside the ball. The headband and VR headset were secured to the participants head to minimize movement artifacts. Participants then followed the auditory instructions provided throughout the experiment from the VR headset. B. Experimental Design Each trial consisted of two consecutive phases: eyes open with fixation (120 s) followed by eyes closed (120 s). The procedure was performed in two stages: on land and on water. EEG acquisition was continuous throughout each trial. Participants were instructed to remain relaxed, avoid unnecessary head or limb movements, and maintain fixation on the VR point during eyes-open periods. Short breaks were provided between trials to prevent fatigue and maintain alertness. C. EEG Processing and Analysis Raw EEG was sampled at 256 Hz and preprocessed using MNE-Python [9]. Data were visually inspected for signal continuity and the presence of artifacts. Signals were band-pass filtered (0.1–30 Hz) to remove slow drifts and high-frequency noise, with an additional 60 Hz notch filter applied to attenuate line interference. Segments with transient movement artifacts were noted but retained for completeness, as median-based summarization reduces their influence. Power spectral densities were estimated from the preprocessed signals, and spectral power was integrated within canonical theta (4–7 Hz) and beta (13–30 Hz) frequency bands. For each trial, the θ/β ratio (TBR) was computed per channel and summarized using the median across electrodes to mitigate PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 21 channel-specific noise. Band-ratio measures were chosen for robustness in mobile EEG contexts [10]. Analyses focused on relative differences in TBR between conditions (land vs. water, eyes-open vs. eyes-closed), consistent with the exploratory aims and limited sample size. D. CO2Safety Monitoring To verify environmental safety, CO2concentration inside the ball was measured at one-minute intervals using an Aranet4 NDIR CO2sensor. Measurements were collected during resting and various activity conditions summarized in Table I. While lying, CO2levels increased linearly and reached approximately 2800 ppm after 18 minutes. Based on observed production rates, extrapolation indicated occupational exposure limits would not be exceeded during the duration of experimental trials [11], [12]. All EEG experiments were conducted well within accepted short-term exposure limits. Participants were continuously monitored for any signs of discomfort or dizziness, and trials were to have been paused if any adverse effects were reported. TABLE I: CO2Production Rates at Varying Activity Levels Location and Activity VCO2 (PPM/min) R2Safe Duration Indoor, resting 98 0.996 61 min Indoor, light exercise 219 0.999 35 min On land, walk/jog 531 0.987 23 min Water, typical balling 595 0.995 22 min Water, vigorous balling 1172 0.997 18 min III. RESULTS Floating on water produced higher θ/β ratios than lying on land across both eye conditions. The largest increase was observed in the on-water, eyes-open condition. On land, TBR values were comparatively lower and showed only modest differences between eyes-open and eyes-closed trials. As summarized in Fig. 2, the effect was the same for both eyes-open and eyes-closed conditions, with TBR consistently higher when participants floated on water compared to lying on land. Differences between eyes-open and eyesclosed conditions were relatively small on land, whereas the floating condition showed a clearer separation between visual states. This suggests that environmental context played a more prominent role than eye condition in shaping the observed TBR patterns. Participants’ self-reports aligned with the EEG findings, describing increased calmness and ease of focus while floating. TABLE II: θ/β Ratio (TBR) Statistics Across Conditions Conditions Mean Std. dev. On water, Eyes closed 10.9 2.21 On water, Eyes open 14.1 3.77 On land, Eyes closed 5.99 0.802 On land, Eyes open 5.32 2.42 Condition-level TBR statistics are summarized in Table II, highlighting overall differences between floating and land environments across eye conditions. on-water (eyes closed) Theta/Beta Ratio (θ/β) on-water (eyes opened) on-land (eyes closed) on-land (eyes opened) Fig. 2: θ/β ratio across conditions. Floating on water is associated with higher TBR, particularly with eyes open. IV. DISCUSSION Despite the exploratory nature of this pilot study, a consistent pattern emerged: floating on water was associated with elevated TBR relative to land-based conditions. Prior work links higher TBR to attentional regulation and reduced stress [5]–[7], supporting the interpretation of State-of-Float as a distinct attentional and affective state. Buoyancy and reduced postural demands may contribute to this effect by lowering physical effort and sensory load. The use of VR allowed sensory conditions to remain constant while the physical environment changed, isolating the contribution of floating itself. V. LIMITATIONS This study involved a small number of participants and was conducted in a dynamic aquatic environment, introducing potential motion artifacts. While preprocessing reduced stationary noise, transient artifacts cannot be fully excluded. Larger studies with additional physiological measures are needed to validate these findings. VI. CONCLUSION We demonstrated that floating inside a water-walking ball is associated with increased θ/β ratios and subjective relaxation compared to land-based conditions. By combining wearable EEG, VR, and environmental safety monitoring, this work establishes a compact framework for studying cognitive and affective states in natural and transitional environments, with potential relevance to XR wellness and reduced-gravity research. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 22 REFERENCES [1] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 [2] S. Mann, “Humanistic intelligence: Ai meets natural intelligence,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2123–2151, 1998. [3] ——, “Wearable computing: Toward humanistic intelligence,” IEEE Intelligent Systems, vol. 16, no. 3, pp. 10–15, 2001. [4] M. Arns, C. K. Conners, and H. C. J. van der Werf, “A decade of eeg theta/beta ratio research in adhd: a meta-analysis,” Journal of Attention Disorders, vol. 17, no. 5, pp. 374–383, 2013. [5] A. R. Clarke, R. J. Barry, and S. J. Johnstone, “Resting state eeg power research in adhd: A review update,” Clinical Neurophysiology, vol. 131, no. 7, pp. 1463–1479, 2020. [6] P. Putman et al., “Eeg theta/beta ratio in relation to attentional control and affective traits,” Frontiers in Psychology, vol. 5, p. 986, 2014. [7] D. van Son et al., “Resting-state frontal alpha asymmetry, theta/beta ratio, and mind wandering,” Annals of the New York Academy of Sciences, vol. 1452, no. 1, pp. 52–68, 2019. [8] “Muse s — eeg mental fitness & sleep headband,” Product page, 2025. [Online]. Available: https://choosemuse.com/pages/muse-s [9] A. Gramfort et al., “Meg and eeg data analysis with mne-python,” Frontiers in Neuroscience, vol. 7, p. 267, 2013. [10] T. Donoghue, J. Dominguez, and B. Voytek, “Electrophysiological frequency band ratio measures conflate periodic and aperiodic neural activity,” eNeuro, vol. 7, no. 6, 2020. [11] Minnesota Department of Health, “Carbon dioxide (co2),” 2024. [Online]. Available: https://www.health.state.mn.us/communities/ environment/air/toxins/co2.html [12] Canadian Centre for Occupational Health and Safety, “Occupational exposure limits,” 2025. [Online]. Available: https://www.ccohs.ca/ oshanswers/hsprograms/occ hygiene/occ exposure limits.html PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 23 Mersivity in Action: How Digital Waters is Bringing Humans, Technology, and Nature Closer Together Jason Spitkoski and Steve Mann, Digital Waters Fig. 1: Digital Waters co-founders, (left-to-right) Steve Mann, Andrew Top, and Jason Spitkoski at Peter Street Basin where our live monitoring solution has helped the Friends of Peter Street Basin (peterstreetbasin.com) do live monitoring for the design of a new aeration and fountain system for installation in the Basin. Abstract—The story of Digital Waters is the coming together of three individuals who share a unified vision shaped by the same urgency: our water is under threat, and the tools we have to understand it simply are not good enough. The three of us were, and still are, driven by a need for immediate action. Here we present an update on our recent progress. Index Terms—Humanistic Intelligence (HI), eXtended Reality (XR), eXtended Intelligence (XI), eXtended Intelligent Reality (XIR), Digital Waters, Peter Street Basin, Water monotoring, Water quality, E. coli I. INTRODUCTION This year, Digital Waters made remarkable progress toward something we believe is foundational to the future of environmental stewardship: Mersivity. Mersivity is the deep reciprocal relationship between humans, their environment, and the technology that connects them to each other and their environment. For us, Mersivity is not just an abstract concept. It is the lived experience of watching people see their environment, particularly water, differently because they can finally see water in a new way. Across Toronto’s creeks, ravines, and lakes, Digital Waters’ low-cost monitoring devices quietly capture hundreds of thousands of images of flowing freshwater, each one the result of illumination by identical internal lighting so that external conditions never distort the story the water is telling. These photographs, simple, honest portraits of a living system, are at the heart of our approach. While most water monitoring relies on dots on a map or numbers buried in spreadsheets, we bring water data directly to people in a way that is visual, tangible, and emotional. A photo invites curiosity. A photo creates empathy. A photo makes the health of our waterways feel real. This year we became the only organization in Canada using underwater water-colour photography as a primary input for water quality assessment. And this approach is paying PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 24 off. Our images have become a powerful training set for our expanding AI (Artificial Intelligence) and XI (eXtended Itelligence) models. These models now detect patterns in our water data that the human eye alone would struggle to quantify. In 2025, we saw a major breakthrough: our AI can infer turbidity, how cloudy or clear the water is, with surprising accuracy using only image data. As our dataset grows, so does the fidelity of our AI, allowing us to move closer to real-time environmental insights that anyone can understand at a glance. Collaboration has been essential in validating this work. Our relationship with the Toronto and Region Conservation Authority (TRCA) deepened this year. With their support, we were able to compare our readings and AI-generated insights against established scientific measurements. This verification didn’t just confirm that our devices work, it improved our models and strengthened the trust we’re building with the broader scientific and conservation community. These partnerships matter because they ensure our tools remain not only innovative, but credible and useful to practitioners who have spent decades caring for these waters. Perhaps most inspiring is the growth of the Digital Waters community itself. We are now a team of more than a dozen volunteers: engineers, designers, field stewards, data enthusiasts, educators, and nature nerds, working side by side to build something Toronto has never had before: a public, continuous, open record of its freshwater health. Every device placed, every photograph captured, and every model trained, is the result of people choosing to give their time and talent to protect the waterways that sustain all life. Mersivity guides everything we do. When humans can see their water, they understand it. When they understand it, they care for it. And when we pair that human connection with good technology and community partnership, we create something powerful: a city where nature is not hidden, but shared. One photo at a time. For the last 3 years, we have been cleaning up Peter Street Basin PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 25 BCI, in which systems infer user state directly from brain activity without explicit input [4]. Recent progress in consumer-grade EEG devices further demonstrates their applicability for tracking attention, stress, and emotional fluctuations in everyday environments [5]. Complementing these developments, research on passive BCI device design provides insights into electrode layout, signal stability, and ergonomic considerations critical for continuous physiological monitoring [6]. Muse Flow builds upon these technological foundations by using wearable EEG as a non-intrusive means to sense changes in the user’s internal cognitive state. C. EEG Neurofeedback and Self-Regulation Neurofeedback research has long explored how EEG-based indicators can support self-regulation of cognitive and emotional states. A substantial body of work links increased Alpha activity to relaxation and meditative states, while elevated Beta activity is associated with focused engagement [7]. More recent efforts integrate neurofeedback with immersive environments such as virtual reality, demonstrating that real-time neural feedback can strengthen self-awareness and improve training outcomes in attention and stress management [8]. Systems developed specifically for meditation training also show that immediate EEG-based feedback helps users transition more effectively into calm or focused mental states [9]. Together, these studies indicate that when external feedback is synchronized with internal physiological changes, users can more easily regulate their cognitive processes. This is central to the interaction paradigm adopted by Muse Flow. D. Audio-Mediated Reality and Physiological-Driven Sound Interaction A substantial body of research has explored the relationship between music, emotion, and physiological responses. Machine-learning models have successfully mapped correlations between musical mood and signals such as EEG or heart rate, demonstrating that auditory stimuli can meaningfully influence internal states [10]. In mixed-reality settings, collaborative music systems have been shown to affect users’ emotional experiences, engagement, and physiological synchrony [11]. Work in natural user interfaces further highlights the role of sound as an intuitive, ambient, and non-disruptive interaction medium. This is exemplified by Mann’s Hydraulophone and Physiphone research in musical expression [12]. These studies collectively support the notion that auditory environments can serve as effective channels for cognitive modulation, providing a foundation for Muse Flow’s use of sound as a form of mediated reality. E. Privacy, Cognitive Sovereignty, and Ethical Considerations in BCI Because EEG data can contain sensitive information (including identity traits, emotional patterns, and cognitive signatures), privacy protection has become a major concern within TABLE I EEG bands and roles in Muse Flow. Band Range (Hz) Symbol Heuristic Role Alpha 8–13 PαRelaxation / calm engagement Beta 14–30 PβTask engagement / cognitive effort BCI research. Systematic reviews warn that unprotected EEG data may enable unintended inference or user profiling. Additional work highlights security vulnerabilities in wearable BCI systems and proposes methods such as secure multiparty computation and signal obfuscation to mitigate these risks. At a conceptual level, Mann’s theories of “sousveillance” and personal data sovereignty argue that individuals, not institutions, should control their own physiological and cognitive information [13], [14]. By performing all signal processing locally within the user’s browser, Muse Flow directly responds to these concerns, aligning with contemporary ethical calls for cognitive autonomy and user-owned data. III. System Overview Muse Flow is implemented as a single-page web application. The high-level pipeline is: 1) EEG Acquisition: Muse streams are accessed over Web Bluetooth. 2) Preprocessing: Artifact clamping, windowing, and channel-wise quality checks. 3) Feature Extraction: FFT-based band-power estimation for α, β 4) State Inference: Calculation of relaxation index. 5) Audio Meditation: Smooth cross-fades, gain shaping, and optional track switching. A. System Architecture Muse Flow is designed as a client-side web application that directly interfaces with Muse EEG headbands via Web Bluetooth. The pipeline includes: 1) EEG Data Acquisition 2) Preprocessing and Artifact Removal 3) Feature Extraction (Alpha and Beta band powers) 4) Real-time Neurofeedback through Audio Modulation B. EEG Data Processing EEG signals are captured at 256 Hz and preprocessed with artifact clamping and Hanning windowing. Power spectral density is calculated using FFT, and Alpha (8–13 Hz) and Beta (14–30 Hz) bands are extracted. Exponential smoothing is applied to reduce noise. C. Biofeedback Logic The Relaxation Index Ridx is calculated as: Ridx =Pα Pα+Pβ PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 32 Fig. 1. Muse Flow architecture. The entire loop runs locally in the browser to minimize latency and preserve data sovereignty. Thresholds are used to determine the system state: •Focus: Ridx <0.3 •Relaxation: Ridx >0.4 Audio fades are applied to prevent abrupt transitions. D. User Interaction and Interface Users experience real-time visualizations of EEG trends and adaptive music feedback. A simulator mode allows testing without hardware. Fig. 2. Home screen of Muse Flow, showing device connection controls and Simulator Mode. IV. Results A. Qualitative Observations During user testing, participants were able to perceive changes in music in response to their cognitive state. The system provided noticeable differences between focus and relaxation states. B. System Performance The local browser implementation allowed low-latency processing of EEG data. No network transmission was required, ensuring privacy. Real-time waveform rendering at 60 FPS provided smooth visual feedback. C. User Experience Feedback Participants reported that the Glassmorphism interface, dynamic backgrounds, and audio modulation contributed to a calming and engaging experience. Users appreciated having immediate feedback on their cognitive state. V. Functional Description Muse Flow operates as a bio-signal processing pipeline that supports participatory personal computing. A. Wearable Interface A Muse EEG headband serves as the sensing component, collecting neural signals from four electrodes. B. Mediated Feedback Loop The application visualizes real-time brainwave data and alters the auditory environment based on physiological state. C. Simulator A built-in signal generator enables testing the logic without requiring the physical EEG hardware. D. Adaptive Modulation The system modulates audio according to two modes: •Focus (High Beta): Music plays when the user exhibits active cognitive engagement. •Relaxation (High Alpha): Music pauses when the user enters a state of deep rest. VI. User Interface Design The interface adopts a Glassmorphism aesthetic using translucent layers and blurred backgrounds to create a calm, low-clutter environment suitable for neurofeedback. A. Dynamic Atmosphere A gradient background animates during the “Flow” (music playing) state and becomes still during the “Relaxation” state, offering peripheral visual cues. B. Real-Time Waveform Rendering Using the HTML5 Canvas API, the system plots Alpha and Beta historical trends with quadratic B´ ezier curves at 60 FPS, producing smooth, organic visualizations. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 33 Fig. 3. Real-time Relaxation Index visualization with Alpha, Beta, and smoothed trend lines displayed during simulator-driven EEG activity. VII. Technical Implementation A. Signal Processing Pipeline The EEG processing pipeline consists of: 1) Acquisition 2) Preprocessing 3) Feature Extraction 4) Analysis B. Data Acquisition & Preprocessing The Web Bluetooth API provides low-latency access to Muse 2 EEG streams (TP9, AF7, AF8, TP10) sampled at 256 Hz. Signals are decoded with muse-js. To ensure data quality: •Amplitudes above 800 µV are clamped to remove artifacts (blinks, jaw clenches). •A Hanning window is applied before FFT to reduce spectral leakage. C. Feature Extraction & Smoothing Using fft.js, the system extracts power in: •Alpha band (8–13 Hz) •Beta band (14–30 Hz) Exponential smoothing stabilizes noisy EEG data: St= (1 −α)St−1+αxt.(1) D. Relaxation Index & Biofeedback Logic The Relaxation Index is defined as: Ridx =Pα Pα+Pβ (2) where Pαand Pβare smoothed spectral powers. State transitions use hysteresis: •Focus: Ridx <0.3 •Relaxation: Ridx >0.4 Audio transitions fade over five seconds to avoid abrupt changes. Fig. 4. Presenting at the Mersivity Symposium, with Muse Flow shown on the projector. VIII. Social Impact A. Data Sovereignty & Privacy Unlike many commercial BCIs, all EEG data is processed client-side in the browser. No neural data is uploaded or stored externally, preserving user autonomy. B. Augmented Mindfulness By externalizing internal neural states, Muse Flow serves as a low-cost neurofeedback mirror, enabling self-regulation and stress awareness. C. Human–Computer Synergy The system demonstrates a design philosophy where computing augments human capability, enhancing focus and rest, rather than distracting or controlling the user. IX. Future Work We see four immediate directions: •Personalized calibration to adapt thresholds to individual EEG baselines. •Richer Audio Mediated Reality including adaptive soundscapes and generative music. •Formal user studies measuring sustained attention, relaxation, and perceived agency •Multi-modal integration to improve robustness. X. Conclusion Muse Flow shows how Humanistic Intelligence principles can be applied using accessible web technologies to create meaningful mind-machine feedback systems. By prioritizing privacy, agency, and cognitive well-being, the system represents a step toward technologies that empower rather than surveil. References [1] S. Mann, “Wearable computing: Toward humanistic intelligence,” IEEE Intelligent Systems, vol. 16, no. 3, pp. 10–15, 2001. [2] ——, “Humanistic computing: ”wearcomp” as a new framework and application for intelligent signal processing,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2123–2151, 1998. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 34 [3] T. R. Mullen, C. A. E. Kothe, Y. M. Chi, A. Ojeda, T. Kerth, S. Makeig, T.-P. Jung, and G. Cauwenberghs, “Real-time neuroimaging and cognitive monitoring using wearable dry eeg,” IEEE Transactions on Biomedical Engineering, vol. 62, no. 11, pp. 2553–2567, 2015. [4] R. Portillo-Lara, B. Tahirbegi, C. A. R. Chapman, J. A. Goding, and R. A. Green, “Mind the gap: State-of-the-art technologies and applications for eeg-based brain–computer interfaces,” APL Bioengineering, vol. 5, no. 3, p. 031507, 2021. [5] M. S. Al-Quraishi, J. Al-Ghamdi, S. Al-Bander, F. Al-Nuaimi, and H. Ellethy, “Recent progress in wearable brain–computer interface devices based on electroencephalogram (eeg) for medical applications: A review,” Sensors, vol. 20, no. 21, p. 6008, 2020. [6] M. A. Lopez-Gordo, D. Sanchez-Morillo, and F. P. Valle, “Dry eeg electrodes,” Sensors, vol. 14, no. 7, pp. 12 847–12 870, 2014. [7] N. Omejc, B. Rojc, P. P. Battaglini, and U. Marusic, “Review of the therapeutic neurofeedback method using electroencephalography: Eeg neurofeedback,” Bosnian Journal of Basic Medical Sciences, vol. 19, no. 3, pp. 213–220, 2019. [8] L. Castanho et al., “The efficacy of virtual reality-based eeg neurofeedback in health-related symptom relief: A systematic review,” Applied Psychophysiology and Biofeedback, 2025. [9] A. E. Nieto-Vallejo, O. F. Ram´ ırez-P´ erez, L. E. Ballesteros-Arroyave, and A. Arag´ on, “Design of a neurofeedback training system for meditation based on eeg technology,” Revista Facultad de Ingenier´ ıa, vol. 30, no. 55, p. e12489, 2021. [10] A. Garg, V. Chaturvedi, A. B. Kaur, V. Varshney, and A. Parashar, “Machine learning model for mapping music mood and human emotion based on physiological signals,” Multimedia Tools and Applications, vol. 81, no. 4, pp. 5137–5177, 2022. [11] R. Schlagowski, D. Nazarenko, Y. Can, K. Gupta, S. Mertes, M. Billinghurst, and E. Andr´ e, “Wish you were here: Mental and physiological effects of remote music collaboration in mixed reality,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023. [12] S. Mann, “Natural interfaces for musical expression: Physiphones and a physics-based organology,” in Proceedings of the 2007 Conference on New Interfaces for Musical Expression (NIME07), 2007. [13] ——, “Declaration of veillance,” [Online]. Available: http://wearcam.org/veillance/, 2013. [14] S. Mann, M. Minsky, and R. Kurzweil, “Society of intelligent veillance,” [Online]. Available: http://wearcam.org/societyofintelligentveillance.htm, 2013. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 35 Let X∈IReality Steve Mann, Hao łMaxž Lv, Mustafa Can Gursesli, Oğuz ‘Oz’ Buruk, and Juho Hamari Figure 1: XR (eXtended Reality) / XI (eXtended Intelligence) is a broad umbrella term that also interpolates between AR, VR, MR, PR, DR, IR, ... AI, IoT, Metaverse, etc., as well as eXtrapolates beyond them. Abstract The big letter łXž is like a mathematical variable that serves as a placeholder for all the łRealitiesž like VR (Virtual Reality), AR (Augmented Reality), Mediated or Mixed (MR), Diminished Reality Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from p[email protected]. , , © 2025 Copyright held by the owner/author(s). Publication rights licensed to Mersivity. WEARTECH™, Weartech Lab™, Mersivity, and WaterHCI™ https://Mersivity.com (DR = tech like dark sunglasses or welding helmets), Digital Reality, Intelligent or Integral or Internet Reality (IR), as well as IoT, DigiTwin, Metaverse, Spatial Computing, and of course AI. Let X∈IR, i.e. let X be any and all of them! But XR is more than just a broad umbrella term for the łalphabet soupž of realities. It also allows us to interpolate between them and eXtrapolate (eXtend) beyond them! Keywords XR (eXtended Reality), VR (Virtual Reality), AR (Augmented Reality), MR (Mediated Reality), PR (Physical Reality), DR (Diminished Reality), SR (Spatial Reality), IR (Intelligent Reality), Metaverse, Mersivity, Spatial Computing, Digital Twin, Metaverse, Lightspace, Hologram, Margoloh, Lightfield PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 36 , , Reference Format: Steve Mann, Hao łMaxž Lv, Mustafa Can Gursesli, Oğuz ‘Oz’ Buruk, and Juho Hamari . 2025. Let X∈IR eality . In .Mersivity-2025, Toronto, Ontario, Canada, 4 pages. https://Mersivity.com XR (eXtended Reality) was invented more than 50 years ago, and named more than 30 years ago, but it is finally catching on today in the world of consumer electronics technologies, as the unifying framework for the Realities, Verses, Intelligences, and łInternetsž. Products like Meta Quest, Apple Vision Pro are great examples. And now we’re beginning to see true XR capabilities in the form of an XR eyeglass that runs an operating system built from the ground-up specifically for XR, namely the Samsung Galaxy XR, running Google’s Android XR operating system. Finally, after more than 50 years, XR has come of age. 1 “Alphabit”-Soup Integration of physical, virtual, and social worlds has led to an łalphabet soupž of łRealitiesž such as Virtual Reality (VR), Augmented Reality (AR), Mediated (or Mixed) Reality (MR), Diminished (and also Digital) Reality (DR), Intelligent (and also Integrated or Internet) Reality (IR), and verses (metaverse, holoverse, omniverse), spatial computing, and other emerging paradigms. The concept of eXtended Reality (XR) is more than just a broad umbrella term to encompass all of these. It also interpolates between them, giving us any combination, providing us with a unifying framework. Moreover, XR takes us on a journey of discovery and invention of new holistic creations, devices, and systems that are conceptually both broad and deep, eXtrapolating beyond the Realities, Verses, Computations, Spaces, etc.. XR also embodies XI (eXtended Intelligence) and is therefore sometimes written XR/XI or XIR (eXtended Intelligent Reality), stylized as XIR , i.e. the capital letter łXž , followed by the Blackboard Bold łRž that represents the set of real numbers (denoted by the letter łIž, followed by the letter łRž in very close proximity to one another, as a single character). In this sense, we can see the letter łXž as a mathematical variable, as in łLet X be the set of all Realitiesž, perhaps even writing it like an equation: X∈IR.(1) This statement (mathematical or otherwise) includes, of course, Intelligent Reality such as physical AI or sustainable AI. XR/XI is a concept that has been around for more than 50 years conceptually, e.g. as a way of seeing in the ultravoilet, infrared, electromagnetic, and acoustic spectrum, including the ability to see radio waves, sound waves, and metavision (sensing sensors and sensing their capacity to sense, i.e. the predecessor of the metaverse), and share these visions with others since XR is shared and collaborative by its very nature [ 9 , 16 , 17 , 23 , 24 ]. The terms eXtended Reality and eXtended Intelligence have been around for more than 30 years [ 1 , 4 , 5 , 7 , 12 , 17 , 18 , 26 ], but are finally gaining widespread traction with projects such as Android XR and Samsung Galaxy XR. 2 DR as a form of XR Technologies that attenuate or diminish our senses like ear plugs, dark sunglasses, welding helmets, baseball caps, etc, are evolving to adaptive noise cancellation, smart sunglasses, and smart welding helmets. All of them, whether smart or not, are examples of DR (Diminished Reality) [2, 3, 6, 8, 13, 15, 19, 21, 25]. Whereas others such as Milgram [ 20 ] and Skarbez [ 27 ] have proposed the idea of a reality continuum, the original XR concept provides a continuum extending from a specific origin or zero point, defined by diminished reality, adding clarity by creating the XR continuum [2, 3, 6, 8, 13, 15, 19, 21, 25]. 3 XR as Wearable AI A good example of XR is Wearable AI [ 14 , 17 ] (Wearable Artificial Intelligence), which can take the form of an AI-driven smart eyeglass that can eXtended our sensory capacity and intelligence [ 17 , 22 ]. Wearable XR has the power to bring technology with us, wherever we go. Wearable XR is quite distinct from other work such as Ivan Sutherland’s device that was used while sitting down in an indoor setting [28]. 4 XR is social! As another example of XR, consider Sicherheitsglaeser, a live performance presented at Ars Electronica Sept. 1997, based on concepts presented the conference [ 10 ], followed by an art exhibit at List Visual Arts Centre Oct-Dec 1997 [ 11 ]. It comprised of a wearable computer supporting a large remote audience of more than 30,000 people, engaged in an XR (eXtended Reality) metavision experience, while at the same time featuring both an inwards-facing display for the wearer, and an outwards-facing display for a local audience. In this way Sicherheitsglaeser facilitated direct interaction between remote and local participants. This example showed how XR can be a highly social medium, as well as a socio-cyber-physical phenomenon that connects us to each other and to our environment. 5 Human-Computer Interaction in the Age of XR and XIR The increasing range of immersive technologies has led to a fractured Human-Computer Interaction (HCI) domain, with modalities such as VR-based interaction, AR usability, and mixed reality. The term "XR" as a unifying variable that bridges realities has emerged. Similarly, HCI requires an integrative framework that treats interaction as a continuous, adaptive relationship between humans, environments, and intelligent systems, rather than as device-bound. In an XR-oriented HCI framework, interactions should be understood as spanning physical, virtual, diminished, and intelligent realities, rather than being confined to a single point on the reality-virtuality axis. The reality continuum largely grounds the understanding of extended reality in the visual modality, treating transparent access to the łrealž as the baseline. Adopting a multi-dimensional perspective rather than a linear spectrum can help integrate multimodal inputs (e.g., gaze, physiology, movement, and affect) with adaptive system responses by letting us evaluate the outcomes of users’ actions and behaviours across different realities. This supports interaction models that can shift dynamically across levels of immersion, augmentation, and mediation. This approach is relevant to contemporary HCI challenges (also, in this sense affective computing can be included), as users now interact with environments PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 37 Let X∈IReality , , that sense, infer, and respond to human states in real time, rather than just interfaces. Furthermore, XR changes the way we view HCI by framing it as an inherently social and socio-cyber-physical phenomenon. As wearable and spatial computing systems increasingly support shared, co-located, and remote experiences, interaction extends beyond individual usability towards collective meaning-making, collaboration, and behavioural influence. For example, wearables, beyond their functional value of enabling technologies to be carried on and embedded in the body, also act as cultural mediators that are integrated into users’ expressions of identity, e.g., through fashion design. As such, they can be devices representing the multiple identities users may adopt across differently manifested realities. When examined across multiple dimensions of individual experience, societal organization, and cultural meaning-making, a linear reality continuum, primarily grounded in the visual modality, falls short of capturing the richness of XR experiences and their broader affordances. Supporting this perspective, the concept of XIR further broadens HCI by embedding intelligence directly into immersive environments, where interaction is shaped by continuous sensing, inference, and adaptation. XRand XIR-based HCI systems thus function not only as interfaces, but also as mediators of social presence, emotional regulation, and cognitive engagement, actively interpreting human states and contextual cues to support adaptive and anticipatory interactions across contexts. 6 XIR an the Role of Legal Terminology and Conceptual Precision Although eXtended Reality (XR) has gained increasing recognition as an integrative concept for a broad range of immersive technologies, the notion of eXtended Intelligent Reality (XIR) should at present be understood as a conceptual proposal rather than an established category. From a legal-theoretical perspective, this distinction is significant. Law does not regulate technologies as such, but rather stabilizes expectations through clearly defined categories that allocate rights, obligations, and responsibilities. The emerging convergence of immersive environments with adaptive, AI-driven functionalities challenges existing legal vocabularies, which were largely developed for discrete software systems, passive media, or clearly identifiable automated decision-making processes. In the absence of terminological precision, XIR-like systems risk being classified inconsistently across regulatory frameworks, thereby undermining legal certainty and accountability. This paper therefore argues that the scientific development of XIR must be accompanied by a deliberate effort toward conceptual clarification, not as a normative exercise, but as a prerequisite for meaningful empirical research, interdisciplinary communication, and future regulatory alignment. Moreover, law itself tends to be a form of oversight (surveillance), especially in the context of law enforcement. Along with surveillance, we have the emergence of sousveillance (łundersightž), a more distributed form of participatory łveillancež that can keep the law in check. We could say that łThe problem with the law is that so many stupid laws are enacted at such a fast-pace, that technology is having a hard time catching up.ž and sousveillance is already helping to keep overzealous police in check as we build a world in which the law itself is held accountable. 7 Conclusion XR is a unifying framework for AR, VR, MR, PR, DR, IR, ..., AI, IoT, Metaverse, Holoverse, Omniverse, Digital Twin, and Spatial Computing. We should embrace XR as a way to navigate through the otherwise fragmented universe of Realities, Verses, Intelligences, and Internets, and perhaps clear out some of the clutter in a world with too much jargon. 8 Acknowledgements References [1] Laura Boffi and Gianluca Boccia. 2024. The Co-Drive Service: Probing into Preliminary Social Outcomes through Early-Stage Prototyping. (2024). [2] Siru Chen, Lingxin Yu, Yuxuan Liu, Zhifei Ding, Jiacheng Zhang, Xinyue Wang, Jiahao Han, and Richen Liu. 2024. Diminished reality techniques for metaverse applications: A perspective from evaluation. IEEE Internet of Things Journal (2024). [3] Yi Fei Cheng, Hang Yin, Yukang Yan, Jan Gugenheimer, and David Lindlbauer. 2022. Towards understanding diminished reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1ś16. [4] Abdulmotaleb El Saddik, Fabrizio Lamberti, Steve Mann, Filippo Gabriele Pratticò, Ruck Thawonmas, and Yu Yuan. 2024. Metaverse and eXtended uniVerse (XV): Opportunities and Challenges for Consumer Technologies. IEEE Consumer Electronics Magazine (2024). [5] Sophie Foster, Larissa Barth, and Zaryab Chaudhry. 2024. Virtual Gathering Platforms in Academic Teaching: Potential and Applications. Electronic Journal of e-Learning 22, 3 (2024), 124ś140. [6] Jan Herling and Wolfgang Broll. 2012. Pixmix: A real-time approach to highquality diminished reality. In 2012 ieee international symposium on mixed and augmented reality (ismar). IEEE, 141ś150. [7] Peter Hoffmann. 2024. Das Verschmelzen von Welten und... versen. In Next Generation Internet: Die Verschmelzung von Realität und Virtualität im Metaversum. Springer, 27ś86. [8] Norihiko Kawai, Tomokazu Sato, and Naokazu Yokoya. 2015. Diminished reality based on image inpainting considering background geometry. IEEE transactions on visualization and computer graphics 22, 3 (2015), 1236ś1247. [9] S. Mann. 1992. Wavelets and "Chirplets": TimeśFrequency "Perspectives" With Applications. In Advances in machine vision: strategies and applications. World Scientific, 99ś128. [10] Steve Mann. 1997. Humanistic Intelligence (H.I.). Proceedings of Ars Electronica (Sep 8-13 1997), 217ś231. Invited plenary lecture, Sep. 10, http://wearcam.org/ars/ http//www.aec.at/fleshfactor, Republished in: Timothy Druckrey (ed.), Ars Electronica: Facing the Future, A Survey of Two Decades, MIT Press, pp 420ś427. [11] Steve Mann. 1997. Sicherheitsglaeser. Public performance at Ars Electronica (Sep 8-13 1997). See also http://wearcam.org/ars/ and http://wearcam.org/lvac/ where it was also exhibited at List Visual Arts Centre, Oct9 - Dec28, 1997. [12] Steve Mann. 2001. Can humans being clerks make clerks be human?śexploring the fundamental difference between ubicomp and wearcomp (können menschen, die sich wie angestellte benehmen, angestellte zu menschlichem verhalten bewegen? zum fundamentalen unterschied zwischen ubicomp und wearcomp). it-Information Technology 43, 2 (2001), 97ś106. [13] Steve Mann and Woodrow Barfield. 2003. Introduction to mediated reality. International Journal of Human-Computer Interaction 15, 2 (2003), 205ś208. [14] Steve Mann, Li-Te Cheng, John Robinson, Kaoru Sumi, Toyoaki Nishida, Soichiro Matsushita, Ömer Faruk Özer, O ˇ guz Özün, C. Öncel Tüzel, Volkan Atalay, A. Enis Çetin, Joshua Anhalt, Asim Smailagic, Daniel P. Siewiorek, Francine Gemperle, Daniel Salber, Sam Weber, Jim Beck, Jim Jennings, and David A. Ross. May/June 2001. Wearable AI. Intelligent Systems, IEEE 16, 3 (May/June 2001), 1ś53. [15] Steve Mann and James Fung. 2002. EyeTap devices for augmented, deliberately diminished, or otherwise altered visual perception of rigid planar patches of real-world scenes. Presence 11, 2 (2002), 158ś175. [16] Steve Mann, Mark Mattson, Steve Hulford, Mark Fox, Kevin Mako, Ryan Janzen, Maya Burhanpurkar, Simone Browne, Craig Travers, Robert Thurmond, et al . 2021. Water-Human-Computer-Interface (WaterHCI): Crossing the borders of computation clothes skin and surface. Proceedings of the 23rd annual WaterHuman-Computer Interface Deconference (Ontario Place TeachBeach, Toronto, Ontario, Canada). Ontario Place TeachBeach, Toronto, Ontario, Canada (2021), 6ś35. [17] Steve Mann and Charles Wyckoff. 1991. Extended Reality. MIT 4-405, http://wearcam.org/xr.htm (1991). [18] Steve Mann, Yu Yuan, Fabrizio Lamberti, Abdulmotaleb El Saddik, Ruck Thawonmas, and Filippo Gabriele Prattico. 2023. eXtended meta-uni-omni-Verse (XV): Introduction, Taxonomy, and State-of-the-Art. IEEE Consumer Electronics Magazine 13, 3 (2023), 27ś35. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 38 , , [19] Siim Meerits and Hideo Saito. 2015. Real-time diminished reality for dynamic scenes. In 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops. IEEE, 53ś59. [20] Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321ś 1329. [21] Shohei Mori, Sei Ikeda, and Hideo Saito. 2017. A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications 9 (2017), 1ś14. [22] Monique Morrow, Jay Iorio, Greg Adamson, BC Biermann, Katryna Dow, Takashi Egawa, Danit Gal, Ann Greenberg, John C. Havens, Dr. Sara R. Jordan, Lauren Joseph, Ceyhun Karasu, Hyo eun Kim, Scott Kesselman, Steve Mann, Preeti Mohan, Lisa Morgan, Pablo Noriega, Dr. Stephen Rainey, Todd Richard, Skip Rizzo, Francesca Rossi, Leanne Seeto, Alan Smithson, Mathana Stender, and Maya Zuckerman. 2019-2020. The IEEE global initiative on ethics of autonomous and intelligent systems. Extended Reality in A/IS (2019-2020), 1ś29. [23] Florian ‘Floyd’ Mueller, Maria F Montoya, Sarah Jane Pell, Leif Oppermann, Mark Blythe, Paul H Dietz, Joe Marshall, Scott Bateman, Ian Smith, Swamy Ananthanarayan, Ali Mazalek, Alexander Verni, Alexander Bakogeorge, Mathieu Simonnet, Kirsten Ellis, Nathan Arthur Semertzidis, Winslow Burleson, John Quarles, Steve Mann, Chris Hill, Christal Clashing, and Don Samitha Elvitigala. 2024. Grand challenges in WaterHCI. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1ś18. [24] Sarah Jane Pell, Steve Mann, and Michael Lombardi. 2023. Developing WaterHCI and OceanicXV technologies for diving. In OCEANS 2023-Limerick. IEEE, 1ś10. [25] Sanni Siltanen. 2017. Diminished reality for augmented reality interior design. The Visual Computer 33 (2017), 193ś208. [26] Uğurluer Simge and Seven Mert. 2024. İngilizce iletişim çalışmalarında genişletilmiş gerçeklik araştırma eğilimlerinin bibliyometrik bir analizi: Genişletilmiş gerçeklik teknolojilerine gelişen rağbetin haritalanması. Connectist: Istanbul University Journal of Communication Sciences 66 (2024), 147ś181. [27] Richard Skarbez, Missie Smith, and Mary C Whitton. 2021. Revisiting Milgram and Kishino’s reality-virtuality continuum. Frontiers in Virtual Reality 2 (2021), 647997. [28] I. Sutherland. 1968. A head-mounted three dimensional display. In Proc. Fall Joint Computer Conference. Thompson Books, Wash. D.C., 757ś764. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 39 XR/XI (Extended Intelligence/Reality) Light Paper Steve Mann, Bernard Kress, and Bharath Rajagopalan Figure 1: Lightspace Principle showing example of eyeglass formed by four optical elements: (1) Camera; (2) Aremac; (3) Neseye; and (4) Eyesensor. These four elements facilitate XR (eXtended Reality) which is a broad umbrella term that interpolates between AR, VR, MR, ... AI, IoT, Metaverse, etc., as well as eXtrapolates beyond them. Abstract We present a łLight Paperž (white paper, red paper, green paper, blue paper, yellow, cyan, magenta, black, etc., over the full range of light from 0 to maximum) on XR (eXtended Reality) / XI (eXtended Intelligence). XR is more than just a broad umbrella term for the łalphabet soupž of realities (AR, VR, MR, IoT, DigiTwin, Metaverse, etc.). It also allows us to interpolate between these various realities and eXtrapolate (eXtend) beyond them. Most importantly, we present the four possible fundamental optical (or otherwise) elements of an XR system in the context of XR as a way of inventing, designing, building, and deploying a wide-range of devices, systems, and creations pertaining to eXtending the human senses and intellect. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from p[email protected]. , , © 2025 Copyright held by the owner/author(s). Publication rights licensed to Weartech Lab™, Mersivity™and WaterHCI™. Mersivity-2025 https://Mersivity.com Keywords XR (eXtended Reality), VR (Virtual Reality), AR (Augmented Reality), MR (Mediated Reality), PR (Physical Reality), DR (Diminished Reality), SR (Spatial Reality), IR (Intelligent Reality), Metaverse, Mersivity, Spatial Computing, Digital Twin, Metaverse, Lightspace, Hologram, Margoloh, Lightfield Reference Format: Steve Mann, Bernard Kress, and Bharath Rajagopalan. 2025. XR/XI (Extended Intelligence/Reality) Light Paper . In .IEEE, New York, NY, USA, 4 pages. https://Mersivity.com 1 “Alphabit”-Soup Integration of physical, virtual, and social worlds has led to an łalphabet soupž of łRealitiesž such as Virtual Reality (VR), Augmented Reality (AR), Mediated (or Mixed) Reality (MR), Diminished (and also Digital) Reality (DR), Intelligent Reality (IR), and verses (metaverse, holoverse, omniverse), spatial computing, and other emerging paradigms. The concept of eXtended Reality (XR) is more than just a broad umbrella term to encompass all of these. It also interpolates between them, giving us any combination, providing us with a unifying framework. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 40 , , Moreover, XR takes us on a journey of discovery and invention of new holistic creations, devices, and systems that are conceptually both broad and deep, eXtrapolating beyond the Realities, Verses, Computations, Spaces, etc.. XR also embodies XI (eXtended Intelligence) and is therefore sometimes written XR/XI or XIR (eXtended Intelligent Reality), stylized as XIR , i.e. the capital letter łXž , followed by the Blackboard Bold łRž that represents the set of real numbers (denoted by the letter łIž, followed by the letter łRž in very close proximity to one another, as a single character). In this sense, we can see the letter łXž as a mathematical variable, as in łLet X be the set of all Realitiesž, perhaps even writing it like an equation: X∈IR.(1) This statement (mathematical or otherwise) includes, of course, Intelligent Reality such as physical AI or sustainable AI. XR/XI is a concept that has been around for more than 50 years conceptually, e.g. as a way of seeing in the ultravoilet, infrared, electromagnetic, and acoustic spectrum, including the ability to see radio waves, sound waves, and metavision (sensing sensors and sensing their capacity to sense, i.e. the predecessor of the metaverse), and share these visions with others since XR is shared and collaborative by its very nature [ 9 , 16 , 17 , 23 , 24 ]. The terms eXtended Reality and eXtended Intelligence have been around for more than 30 years [ 1 , 4 , 5 , 7 , 12 , 17 , 18 , 26 ], but are finally gaining widespread traction with projects such as Android XR and Samsung Galaxy XR. 2 XR is Holomargolography We proffer a generalized embodiment of XR comprised of four fundamental elements: (1) A camera that senses our surroundings. This can be a single camera, or an array of cameras, or a even a holographic video camera; (2) An aremac that presents us with our surroundings. This can be a single display element, or an array of displays, including one or more holographic video displays. It can even be a contraption of sorts that operates as a reciprocal of a camera in the sense that it functions like a projector with features usually found only in cameras not projectors. Thus the aremac might have an f-stop in it that controls the depth-of-field of displayed content as if it were a camera operating in reverse. The role of the aremac is reciprocal to the role of the camera. The etymology of the word łaremacž is łcameraž spelled backwards [14]; (3) An EyeSensor (abbreviated to łEyeSenž) that senses your gaze, postion, focus, etc., of the eye. Ideally this is more than just an eye tracker. Ideally it also senses at what depth the lens(es) of your eye(s) is/are focused; (4) A Neseye, which is an outwards-facing display. This outwardsfacing display shows your eyes to others. The Neseye is the reciprocal of the EyeSensor. The etymology of Neseye is Eyesen spelled backwards. Not all systems have or even require these four elements. For example, a person wearing a transparent eyeglass does not require an Eyesen, since their eyes are visible to others through the clear glass. Some devices do not even have a camera, e.g. we might have an eyeglass or device that simply displays text to the wearer. An example of such a device is the Reflection Technologies Private Eye which has no camera, and can only display a red screen, typically of text. Some devices might not have a display, or the łdisplayž may be non-visual. For example, a blind person may wear an eyeglass camera that łdisplaysž to earphones. Thus we shall consider the content of Fig 1 as a general framework for a wide range of possible devices and systems that currently exist, or are yet to be invented. It is useful to consider an Outer Glass (OG) that houses one or more cameras, and possibly the Neseye, as well as an Inner Glass (IG) that houses the Aremac and Eyesen. The Inner Glass faces the wearer, and the Outer Glass faces away from the wearer. Between these is a processor, which we simply refer to as XR. Typically the processing of XR exists between the Camera and the Aremac, processing rays of light that travel from left (Subject) to right (Eye). If present, the Eyesensor may assist this process, and/or also be used to feed a rendering process in the Neseye. Any of the four elements can be absent, e.g. we may have a VR (Virtual Reality) headset that has no camera and has no connection to, or ability to sense, the physical world. In fact all four elements could be missing, in which case we simply have PR (Physical Reality). Technologies that attenuate or diminish our senses like ear plugs, dark sunglasses, welding helmets, baseball caps, etc, are evolving to adaptive noise cancellation, smart sunglasses, and smart welding helmets. All of them, whether smart or not, are examples of DR (Diminished Reality) [ 2 , 3 , 6 , 8 , 13 , 15 , 19 , 21 , 25 ]. Whereas others such as Milgram [ 20 ] and Skarbez [ 27 ] have proposed the idea of a reality continuum, the original XR concept provides a continuum extending from a specific origin or zero point, defined by diminished reality, adding clarity by creating the XR continuum [ 2 , 3 , 6 , 8 , 13 , 15, 19, 21, 25]. 3 XR as Wearable AI A good example of XR is Wearable AI [ 14 , 17 ] (Wearable Artificial Intelligence), which can take the form of an AI-driven smart eyeglass that can eXtended our sensory capacity and intelligence [ 17 , 22 ]. Wearable XR has the power to bring technology with us, wherever we go. Wearable XR is quite distinct from other work such as Ivan Sutherland’s device that was used while sitting down in an indoor setting [28]. 4 Inner Glass and Outer Glass As another example of XR, consider Sicherheitsglaeser, a live performance presented at Ars Electronica Sept. 1997, based on concepts presented the conference [ 10 ], followed by an art exhibit at List Visual Arts Centre Oct-Dec 1997 [ 11 ]. It comprised of a wearable computer supporting a large remote audience of more than 30,000 people, engaged in an XR (eXtended Reality) metavision experience, while at the same time featuring both an inwards-facing display for the wearer, and an outwards-facing display for a local audience. In this way Sicherheitsglaeser facilitated direct interaction between PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 41 For heart-rate sensing, we use the COOSPO H808S cheststrap monitor, which supports BLE and ANT+ and streams real-time heart-rate data to compatible devices and fitness applications [12], [13]. The chest strap offers robust measurements even during movement and is designed for sports and exercise use. III. SYSTEM DESIGN A. Overall Architecture Fig. 1 shows the overall data flow of our system, which bridges BLE sensing and standalone VR rendering: •The user wears the H808S chest strap, which broadcasts HR over BLE. •A PC or Mac receives HR data using a Python script based on the bleak library. •The script updates a local text file (heart.txt) with the latest BPM value. •Using ADB, the script repeatedly pushes heart.txt into the Quest 2’s application sandbox. •A Unity application running on the Quest 2 reads the file periodically and updates the VR scene accordingly. B. Hardware Setup Our prototype uses the following hardware: •COOSPO H808S chest-strap heart-rate monitor: provides real-time heart-rate data over BLE and ANT+ [12]. •Meta Quest 2 VR headset: standalone 6DoF VR display where the meditation environment runs [10], [11]. •PC / Mac: runs Unity editor for development and Python for BLE data collection and ADB file transfer. The heart-rate strap is paired only with the PC/Mac, not directly with the Quest 2. The Quest is connected via USB to enable ADB communication. C. Software Components The software stack consists of three main components: 1) Python BLE + ADB script: implemented using bleak to subscribe to the standard Heart Rate Measurement characteristic, parse BPM, and write it into heart.txt. A background thread repeatedly issues: adb push heart.txt /sdcard/Documents/heart.txt ensuring the file on the Quest 2 is kept up to date. 2) Unity heart-rate reader: a C# script inside the Unity project reads heart.txt from /sdcard/Documents/heart.txt (or from Application.persistentDataPath if using the app sandbox), parses the BPM as an integer, and stores it in a global static variable that other scripts can access. 3) Environment controller: a central controller script maps the current BPM to visual and auditory parameters: skybox color, light intensity, particle emission rate, particle speed, and audio source volumes. Fig. 1: System workflow from COOSPO H808S sensing, Python BLE parsing, ADB transfer, and Unity-based VR adaptation. D. Heart Rate Zones and Environment Mapping To make the feedback intuitive, we designed a multizone mapping between HR and environmental state. Table I summarizes the mapping. Each zone corresponds to a different combination of: •Skybox: dark blue/gray for storms, progressively brighter and warmer toward low HR. •Lighting: reduced intensity for high HR, increasing as HR decreases. •Particles: rain (heavy/light), leaves, and petals with different emission rates and speeds. •Audio: heavy rain, light rain, wind, and calm ambient tracks cross-faded based on HR. By designing overlapping ranges (e.g., 80–85 bpm and 70– PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 48 TABLE I: Heart-rate zones and corresponding VR feedback. HR (bpm) State VR Feedback >100 High stress Heavy rain, very dark storm sky, strong wind, loud storm audio. 95–100 Stress Medium rain, slightly darkened sky, noticeable wind, rain+wind audio. 85–95 Mild stress Light rain, cloudy sky, gentle rainfall and light wind. 80–85 Transition Falling leaves, calm soft wind, light wind ambience. 70–80 Transition Leaves and sakura petals, calmer wind, light wind ambience. 60–70 Relaxed Cherry-blossom petals, bright sky, soft ambient wind. <60 Deep calm Warm sunlight, minimal particles, calm ambient track plus birdsong. 80 bpm as transition bands), we avoid abrupt changes and create a continuous feeling of moving from storm to serenity. IV. UNITY SCENE AND IMPLEMENTATION A. Meditation Environment Layout The VR scene is a simple natural environment intended to minimize distraction and support relaxation. It includes: •A flat grass terrain with a few trees and rocks. •A skybox material whose _SkyTint and _GroundColor are controlled dynamically. •A primary directional light representing the sun. •Three particle systems: rain, leaves, and cherry-blossom petals. •A head-locked heart-rate HUD rendered using TextMesh Pro, displaying current BPM in the corner of the user’s view. B. Dynamic Visual Effects The central controller script reads the heart rate and computes a normalized value: t= InverseLerp(minHR,maxHR,BPM), which is then used to interpolate between visual states. For example, sky color is updated as: Csky = (1 −t)·Cbright +t·Cdark, with additional thresholds used for rain, leaves, and petals as described in Table I. Particle emission rates and start speeds Fig. 2: Example Unity scenes from the prototype: (a) high-HR storm scene with heavy rain and dark sky; (b) low-HR calm scene with bright sky and falling petals. are adjusted to make motion visibly slower and sparser as HR decreases. C. Audio Design We use four looping audio sources: •Heavy rain •Light rain •Wind •Calm ambient + birds The volumes of these sources are cross-faded based on HR. For instance, heavy rain and strong wind dominate above 100 bpm, while calm ambient and birds reach maximum volume below 60 bpm. This layered mixing avoids abrupt silence and maintains continuity as users move through heartrate zones. D. Heart Rate HUD To provide immediate feedback, a small head-locked display shows the current BPM: □67 BPM rendered using TextMesh Pro. The HUD is attached to the main camera in Unity so that it remains in a consistent position in the user’s field of view without interfering with the central meditation imagery. V. EVALUATION AND OBSERVATIONS We conducted informal tests with multiple users wearing the COOSPO H808S strap while standing or sitting still in the VR environment. The following aspects were evaluated qualitatively: •Responsiveness: users observed that rainfall intensity, sky brightness, and particle density changed within approximately 0.2–0.5 seconds after noticeable HR changes. •Interpretability: users reported that the storm-to-sun transition metaphor intuitively matched their perception of “stressful” versus “calm” states. •Comfort: the chest strap and Quest 2 headset were comfortable for short meditation sessions (5–15 minutes), and no motion sickness was reported in this static scene. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 49 Fig. 3: Simulated 5-minute HR timeline showing dynamic transitions through stress and calm states. Although we did not perform a controlled user study, these early observations suggest that heart-rate-driven environmental changes are perceptible and can support a feeling of gradual calming when users intentionally slow their breathing and reduce physical tension. VI. DISCUSSION A. Benefits of Heart-Rate-Driven VR Our prototype illustrates several potential benefits of heartrate-aware VR meditation: •Immediate biofeedback: users can see and hear their physiological state reflected in the environment, which may support self-regulation skills similar to traditional HRV biofeedback [3], [6]. •Embodied metaphor: stormy versus calm weather provides an intuitive metaphor for high versus low arousal, making physiological changes more tangible. •Low-cost hardware: the system uses an inexpensive consumer chest strap and a widely available standalone headset. B. Limitations Several limitations should be noted: •No direct BLE in Quest app: we rely on an external PC/Mac and ADB file transfer, which complicates deployment and requires USB connectivity to the headset. •Heart rate only: heart rate alone is a coarse measure; HRV and other signals (e.g., breathing) could provide richer information about stress and relaxation [1], [2]. •No controlled study: we have not yet conducted a formal experiment comparing our system to non-adaptive VR or to traditional 2D biofeedback interfaces. C. Future Work Future extensions of this work include: •Implementing a native Android/BLE plugin for Quest 2 or newer headsets to remove the ADB file-transfer step. •Incorporating breathing exercises and visual breathing guides synchronized with HR or HRV [5]. •Conducting a user study to quantify changes in subjective relaxation, stress, and engagement. •Exploring adaptive guidance, such as voice prompts that react to sustained high HR or rapid decreases. VII. CONCLUSION We have presented a heart-rate-aware VR meditation space that uses real-time physiological data to drive weather, lighting, particles, and audio in an immersive natural environment. Using a COOSPO H808S chest strap, a Python BLE gateway, ADB file transfer, and a Unity application on Meta Quest 2, we demonstrated a complete pipeline from body to virtual world. Our prototype suggests that even a single physiological signal—heart rate—can be leveraged to create engaging, responsive VR experiences that may support relaxation and emotional self-regulation. ACKNOWLEDGMENT We thank the members of MannLab Canada for feedback on the system design and assistance with testing. REFERENCES [1] H.-G. Kim, E.-H. Cheon, D.-H. Bai, Y. H. Lee, and B.-H. Koo, “Stress and heart rate variability: A meta-analysis and review of the literature,” Psychiatry Investigation, vol. 15, no. 3, pp. 235–245, 2018. [2] H. G. Kim et al., “Heart rate variability for evaluating psychological stress: A review,” Neurosignals, vol. 31, no. 4, pp. 187–200, 2023. [3] Y. Pratviel, P. Bouny, and V. Deschodt-Arsac, “Immersion in a relaxing virtual reality environment is associated with similar effects on stress and anxiety as heart rate variability biofeedback,” Frontiers in Virtual Reality, vol. 5, p. 1358981, 2024. [4] R. S. Venuturupalli et al., “Virtual reality–based biofeedback and guided meditation for rheumatology patients with chronic pain: A pilot study,” Journal of Rheumatology, vol. 46, no. 3, pp. 241–243, 2019. [5] L. Chittaro et al., “Virtual reality experiences for breathing and relaxation training with biofeedback,” International Journal of HumanComputer Studies, vol. 180, p. 103123, 2024. [6] J. I. Kerr et al., “The effectiveness and user experience of a biofeedback virtual reality intervention for stress,” BMC Digital Health, vol. 1, no. 1, p. 42, 2023. [7] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 [8] R. Wang et al., “Relationship between heart rate and perceived stress in intensive care residents,” JMIR Formative Research, vol. 8, p. e60759, 2024. [9] S. Ziyadidegan et al., “Quantifying mental stress using cardiovascular responses: A review,” Sensors, vol. 25, no. 14, p. 4281, 2025. [10] “Meta quest 2 specifications,” https://www.meta.com/quest/products/ quest-2/tech-specs/, 2021, accessed: 2025-12-02. [11] “Meta quest 2,” https://en.wikipedia.org/wiki/Quest 2, 2024, accessed: 2025-12-02. [12] “Coospo h808s chest strap heart rate monitor user manual,” https:// manuals.plus/coospo/h808s-heart-rate-monitor-manual, 2021, accessed: 2025-12-02. [13] “H808s chest strap heart rate monitor product page,” https://www. coospo.com/products/h808s-chest-strap-heart-rate-monitor, 2021, accessed: 2025-12-02. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 50 Safeguarding Humanity with Sousveillance: Extending Sensing Inwardly and Outwardly Somin Mindy Lee Dept. Electrical and Computer Eng. University of Toronto Toronto, Canada Aydin Hosseingholizadeh MannLab University of Toronto Toronto, Canada Despina Tzanetakis Div. of Engineering Science University of Toronto Toronto, Canada Nishant Kumar Dept. Electrical and Computer Eng. University of Toronto Toronto, Canada Steve Mann Dept. Electrical and Computer Eng. University of Toronto Toronto, Canada Abstract—As computer vision becomes increasingly embedded in technologies that surround us, ensuring transparency, fairness, accountability, privacy, and ethics in vision-based AI systems is critical for social good. This paper explores the intersection of computer vision, environmentalism, and eXtended Reality (XR) to design ethical, privacy-preserving wearable AI systems. We propose a novel AI-embedded eyeglass that integrates Muse electroencephalogram (EEG) device with vision-based XR to enhance human-AI interaction and environmental perception. A hardware configuration combining eyewear (Vuzix) and EEG (Muse) is explored. Our approach leverages principles of sousveillance (undersight) to empower individuals with real-time AI auditing, environmental awareness monitoring, and neuroadaptive feedback for XR applications. By aligning computer vision with ethical sousveillance and sustainability, we propose a framework for AI-driven wearables that enhance human agency while safeguarding privacy and accountability. I. INTRODUCTION Traditionally, sensing technologies have focused on either the external environment or the internal state of the body. However, a comprehensive understanding of human context may be achieved by integrating both inward-facing and outward-facing sensing modalities. By combining external environmental data with internal physiological signals, such as photoplethysmography (PPG), and EEG from devices such as the Muse headband, we can gain insight into both what is happening around us and how these external factors might affect our internal states [1]. This dual-sensing approach enables context-aware applications that monitor everything from ambient conditions to subtle changes within the body. For example, correlating physiological data with environmental cues can help explain stress responses or other behavioral or health patterns. Such correlations can give us a more nuanced understanding of human health and performance. Moreover, this framework lays the foundation for a balanced interplay between surveillance and its counterpoint, sousveillance. Surveillance denotes the conventional practice of top-down oversight by centralized entities often hidden from view, whereas sousveillance represents a counterbalancing, bottom-up “undersight” approach that empowers individuals to monitor their environment, including the monitors themselves, thereby redefining traditional power dynamics and laying the groundwork for more participatory, ethically driven sensing systems [2], [3]. Fig. 1. Sousveillant system based on Muse-S brain-sensing headband, ”Mind over Motor” [4] Fig. 2. Muse-S together with Vuzix Shield PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 51 In the current era, AI is often associated with machines of surveillance continuously monitoring and interpreting our behaviors [5]–[7]. However, sousveillance, characterized by humanistic intelligence and a bottom-up approach to sensing, serves as a crucial counterbalance. By empowering individuals with tools that are understandable and transparent to the enduser, sousveillance offers an ethical and privacy-preserving alternative to traditional, opaque AI surveillance systems. This paper proposes an integrated wearable system that unifies external vision, through a smart eyeglass-based display system, adding to it, both inwards-facing sensing modalities from devices like an InteraXon Muse brain-sensing headband, as well as outwards-facing sensing modalities like additional cameras so that it has stereo vision in all directions (in addition to the two forwards-facing cameras built into the eyeglass). The goal is to provide a holistic, context-aware sensing system where AI-driven decisions can be interpretable by the user, ensuring that technology remains a tool for enhancing human agency rather than compromising privacy. In doing so, we aim to establish a framework that supports a balanced, human-centric approach to AI and priveillance (privacy/sur/sousveillance) 3. Fig. 3. A conceptual Venn diagram illustrating the relationships among surveillance, sousveillance, privacy, and related constructs such as the panopticon, olig-opticon, equiveillance, and priveillance [8] In particular, it is very important to distinguish between surviellant privacy (panoptic) and sousveillant privacy (oligoptic). For example, social media allows us to hide our information from other users, but not from the company or government officials at the top of the hierarchy. A truly private conversation between two or more individuals allows them to be sensed by each other but not by any government or other “oversight” entity. Privacy is the word used in both cases, but the meanings of the word “privacy” in each case are almost exact opposites. II. RELATED WORKS: CURRENT LANDSCAPE OF WEARABLE SENSING TECHNOLOGIES The evolution of wearable sensing technologies has produced sophisticated systems capable of monitoring a wide range of physiological and environmental parameters. However, research efforts have typically bifurcated into two distinct approaches: inward-facing technologies that monitor internal physiological states and outward-facing technologies that observe the external environment. This division creates a fragmented understanding of human context that an integrated priveillance-based approach could overcome. A. Sousveillance The concept of sousveillance, characterized by humanistic intelligence and a bottom-up approach to monitoring, offers an ethical and privacy-preserving alternative to traditional, opaque AI surveillance systems. However, fully realizing this vision requires wearable systems that can monitor both internal physiological states and external environmental conditions while ensuring that the interpretation and control of this information remains with the user. The technical implementation of sousveillance systems requires efficient object tracking capabilities that maintain privacy while providing accurate environmental awareness. Nam and Han’s work on MDNet demonstrates how deep neural networks can effectively track objects across video frames, providing a foundation for outward-facing sensing in sousveillance systems [9]. Unlike traditional surveillance systems that rely on fixed cameras with centralized control, sousveillance implementations based on wearable cameras benefit from adaptive tracking approaches that maintain user agency and privacy. B. Inward-Facing Sensing Technologies Wearable sensors focused on monitoring internal physiological parameters have seen substantial advancement in recent years. Innovations in biochemical sensing have enabled comprehensive monitoring of multiple biomarkers through noninvasive approaches. A notable example is the development of a flexible dual target electrochemical sensor by Mugo et al. that simultaneously detects pH and cortisol in human sweat using a microneedle design [10]. This technology demonstrates how non-invasive approaches can provide valuable insights into complex biochemical processes occurring within the body. Similarly, dual-function electrochemical sensors capable of detecting multiple analytes such as uric acid and glucose in sweat represent significant progress in multi-parameter physiological monitoring [11]. These sensors feature easy preparation, fast detection, and high sensitivity, making them practical for continuous health monitoring applications. The ability to simultaneously track multiple biomarkers provides a more comprehensive view of internal physiological states than PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 52 single-parameter systems, as it can also improve diagnostic accuracy and facilitate proactive health interventions. Further advancements in inward-facing sensing include the development of paper-based microfluidic sweat sensors with dual-signal readout capabilities. Such systems can simultaneously detect a range of biomarkers including glucose, lactate, uric acid, magnesium ions, pH value, and cortisol through a combination of colorimetric and electrochemical sensing mechanisms [12]. These technologies represent sophisticated approaches to internal physiological monitoring but remain focused exclusively on the body’s internal state without consideration of external environmental factors. Integration of multiple sensing modalities within physiological monitoring is exemplified by a dual-mode wearable sensor that interfaces a hydrogel film with a solid ion-selective electrode [13]. This technology enables the concurrent measurement of heart rate through pressure response and sweat electrolytes, demonstrating how different sensing mechanisms can be combined within a single platform. However, even these multi-modal systems remain confined to internal physiological parameters without engagement with the external environment. Another noteworthy development is a wearable dual-mode sensor that simultaneously monitors electrocardiogram and arterial pulse for cuffless blood pressure measurement [14]. By encapsulating a liquid metal pressure sensing circuit within conductive ionogel, this system eliminates the need for multiple discrete sensors in pulse transit time sensing. While this represents significant progress in simplifying physiological monitoring, it maintains the exclusive focus on internal signals characteristic of inward-facing systems. C. Outward-Facing Sensing Technologies In contrast to inward-focused systems, outward-facing wearable technologies concentrate on monitoring environmental conditions, spatial positioning, or user interactions with the external world. These technologies include ambient sensors, motion detection systems, and spatial awareness tools that provide data about the user’s surroundings and activities rather than their internal physiological state. Research on outward-facing sensing has explored various mechanisms including resistive, capacitive, triboelectric, piezoelectric, thermo-electric, and pyroelectric approaches [15]. Each of these sensing modalities offers unique capabilities for detecting different aspects of the external environment. Capacitive sensing technologies have been employed for detecting interactions with the external environment, such as finger movements and eye blinking. These systems track how the user engages with their surroundings, providing valuable contextual information about physical interactions and movements. Similarly, triboelectric sensors have been integrated into textile-based systems for virtual/augmented reality control, enabling recognition of joint motion and facilitating interaction with digital environments. These outward-facing systems provide valuable information about the user’s surroundings and activities but lack insight into how these external factors affect internal physiological states. This limitation restricts their ability to establish meaningful correlations between environmental conditions and physiological responses, which is essential for comprehensive context awareness. D. The Integration Gap in Current Research Despite significant advancements in both inward and outward sensing technologies, there remains a critical gap in systems that effectively integrate both approaches. Most multifunctional wearable systems combine different sensing modalities within either the inward or outward domain rather than bridging across these domains to create a unified understanding of human context. One notable attempt at integration is the double-sided wearable multifunctional sensing system described by researchers for human-ambience interface [16]. This system addresses the challenge of decoupling interferences from various signals by customizing the pattern and morphology of sensing electrodes and modifying active materials. Through a double-sided partition layout with serpentine interconnections, the system reduces motion artifacts and ensures simultaneous operation of multiple sensing modules. This approach represents an important step toward bridging the gap between inward and outward sensing, as it considers both the perception of ambient changes and the timely feedback of the human body. However, even this system does not fully realize the comprehensive monitoring of both internal physiological states and external environmental conditions in a unified framework that prioritizes human agency and interpretability. The extensive review by Zeng et al. on wearable multifunctional sensing technology highlights the diversity of sensing mechanisms and their applications in healthcare while revealing the limited integration between systems monitoring internal physiology and those monitoring external conditions [15]. This underscores the need for more holistic approaches that provide a complete picture of human context by considering both internal and external factors simultaneously. Future research should focus on developing integrated platforms that not only combine multimodal sensing but also leverage advanced machine learning techniques to adaptively interpret combined data streams in real time, thereby enhancing both the precision and practical applicability of wearable monitoring systems. III. METHODS A. Collecting Brainwaves Brainwaves were collected through the Muse S headband, which were then transmitted through the Muse app on a mobile device. This was then forwarded through the Open Sound Control (OSC) protocol to the UDP port of an ESP32 Xiao S3 Sense. Subsequently, it was transmitted to the wearable smart glasses (Vuzix Shield) via Message Queueing Telemetry Transport (MQTT), enabling real-time display of the brainwaves for the user. A custom android application receives the PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 53 brainwave parameters to be graphed: TP9, AP7, AP8, and TP10 in JSON format. The Muse publishes the data to an MQTT server while the Vuzix Shield glasses subscribe to it. This allows for a data transmission frequency of up to 100 Hz. Prior to visualization, the raw EEG signals are preprocessed using band-pass filtering to minimize noise and remove artifacts, ensuring that only relevant frequency components are plotted. The processed data is then pushed into a queue from which it is visualized in real time. TP9, AP7, AP8, and TP10 values are plotted on a single graph in real time, and they have a range between 0 to 3000. A maximum of 500 data points are displayed on the graph at any given time to ensure high visibility. B. Viewing External Environment Fig. 4. Eight XIAO ESP32 cameras on Vuzix with Muse Camera data is collected through eight XIAO ESP32 cameras, providing a comprehensive field of view. This data is then streamed in real time to a web server, allowing access via local IP addresses. This system is designed to avoid interference with the touch bar on the right of the Vuzix Shield. The camera placement includes the following: •Two cameras on the left side at a distance of 5.3 inches. •Two cameras facing forward, which is part of the Vuzix Shield. •Two cameras facing the rear. •Two cameras facing upwards but angled slightly so that it prevents the user forehead from being in view. This set up ensures efficient data transmission while having a low-latency response. Furthermore, the camera feeds are synchronized with the EEG data, enabling comprehensive temporal correlation between external visual cues and internal brain activity. Current research demonstrates considerable progress in creating multi-functional systems within either the inward or outward sensing domains. However, there is a notable absence of frameworks that effectively integrate both perspectives while maintaining a focus on human agency and interpretability. This gap represents a significant opportunity for developing wearable systems that can provide a more complete understanding of human context and support a balanced, humancentric approach to AI and monitoring. IV. RESULTS In this study, the Muse was successfully integrated with the Vuzix smart glasses and connected to a web server to transmit real-time EEG data, showcasing its potential as a comprehensive tool for concurrently monitoring the internal physiological signals and external environment cues. As seen in Figure 5, two cameras—positioned to capture upward and leftward views—are simultaneously streaming to a web server, showcasing the real-time integration of diverse environmental inputs. Fig. 5. Live Data Streaming from two cameras showcasing an analog oscilloscope and a 24 channel analog mixer. The oscilloscope visualizes the audio signal shaped by the mixer’s adjustments, showcasing real-time waveform modulation. Fig. 6. Digital EEG Signal that is seen on a a local IP address. As shown in Figure 6, the AF7 electrode’s digitized EEG signal is plotted against time, demonstrating the system’s real-time streaming capabilities. The vertical axis represents the raw or scaled amplitude (in arbitrary units), while the horizontal axis shows time in seconds. This setup allows continuous monitoring of brain activity over a web-based interface, enabling immediate feedback and potential integration PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 54 with other sensing modalities. Such a configuration highlights the feasibility of capturing and visualizing neural data in real time, paving the way for advanced applications such as stress detection, cognitive load assessment, and adaptive humancomputer interactions. V. DISCUSSION A. Social and Cultural Impacts Acts of sousveillance, such as individuals recording their interactions in public spaces, allow ordinary people to challenge surveillance systems and assert control over their personal data [17]. For example, using wearable technologies like smartglasses with integrated cameras or EEG sensors, individuals can collect information about their environments while also gaining insights into their internal physiological states. This combination of external and internal sensing fosters a more holistic understanding of the self and surroundings, enabling users to make more informed decisions about their well-being and actions. Sousveillance has the potential to reshape societal norms around privacy, data sharing, and surveillance. It empowers individuals to document their surroundings, offering a counterbalance to institutional surveillance systems by shifting some control back to the public [17]. This challenges traditional hierarchies of power and allows individuals to hold institutions accountable in ways that were previously difficult. On a broader scale, sousveillance democratizes access to AI tools, giving people from diverse backgrounds the chance to contribute to collective knowledge and decision-making [17]. By collecting and analyzing data about their surroundings, individuals can play a more active role in environmental monitoring, public safety, and social accountability, fostering transparency and shared responsibility. Beyond recording, sousveillance encourages individuals to be active participants in shaping their environment. It shifts the focus from control to empowerment, and from oversight to education. With AI-enhanced tools, people can learn more about themselves and their communities, using that knowledge to address personal and societal issues [18]. This shift not only challenges traditional power dynamics but also promotes a more engaged, informed, and empowered society, where people actively shape both their personal futures and the broader systems they live in. B. Ethical Considerations of Sousveillance While surveillance has long been critiqued for infringing on privacy, violating civil liberties, and concentrating power in the hands of institutions, sousveillance has its own risks [19]. The increased power of individuals to gather and share information raises concerns about privacy violations, the potential for harassment or misuse of personal data, and the inadvertent leak of sensitive information that could threaten security. 1) Privacy and Consent: In traditional surveillance systems, there are often established policies and regulations to govern data collection, storage, and usage, even if those policies are not always transparent or equitable. Sousveillance, by contrast, places recording tools in the hands of individuals, complicating the issue of informed consent. Individuals may not always realize they are being recorded, particularly in public or semi-public spaces. This lack of consent undermines the right to privacy and creates potential for abuse [20]. For sousveillance systems to be ethically sound, clear guidelines must be developed to ensure that individuals understand when and how they are being recorded, with mechanisms for opting out where possible. Moreover, the power to record others brings the responsibility to use that power ethically. Users of sousveillance tools must navigate complex social dynamics and cultural norms surrounding privacy. In some cases, sousveillance could exacerbate tensions or lead to the misuse of recordings, such as online harassment [18]. Balancing the potential benefits of sousveillance with the need to protect individuals from harm requires robust ethical frameworks that emphasize accountability, respect for privacy, and informed consent. 2) Data Ownership and Control: Another ethical concern is data ownership. When individuals record and collect data, questions arise about who owns that data and how it should be shared or protected. Should the person who captures the data have full control over it, or should there be limits on what can be done with recordings that involve others? These questions are particularly salient in sousveillance, where personal, environmental, and physiological data may be collected simultaneously [21]. Without appropriate safeguards, there is potential for data misuse, including unauthorized sharing or exploitation of personal information. For example, recordings from wearable devices could be used for surveillance, blackmail, or targeted harassment, undermining the very purpose of sousveillance as a tool for empowerment. To address this, policies that protect data ownership and regulate the sharing of sousveillance recordings are necessary. Additionally, encryption and data anonymization techniques can help mitigate the risks of privacy violations, ensuring that sensitive information is protected from malicious actors. 3) Power Dynamics and Social Inequities: While sousveillance democratizes surveillance and challenges institutional power, it may also reinforce existing social inequalities. Individuals with greater access to technology and resources are more likely to benefit from sousveillance tools, while marginalized groups may face heightened risks. For example, in situations where sousveillance is used to expose abuses of power, vulnerable individuals may be targeted for retaliation or discrimination. Furthermore, the increased visibility of marginalized communities through sousveillance could lead to further surveillance and policing, exacerbating existing forms of social control [22]. In this context, it is essential to consider the ways in which sousveillance could reinforce or challenge social hierarchies. Ethical sousveillance systems must prioritize the protection of vulnerable populations and ensure that the technology is used to empower, rather than oppress, marginalized communities. This includes considering the unintended consequences of PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 55 widespread surveillance capabilities and developing safeguards that prevent the misuse of sousveillance as a tool for harm. 4) Regulatory and Legal Challenges: The legal landscape surrounding sousveillance is still evolving. Existing privacy laws may not adequately address the unique challenges posed by wearable recording technologies and their capacity to capture both public and private information. Current regulations often focus on institutional surveillance, leaving a gap in how sousveillance is governed [3]. Policymakers must consider new regulatory frameworks that balance the rights of individuals to engage in sousveillance with the need to protect the privacy and security of others. This includes establishing clear rules about data ownership, consent, and the ethical use of recordings. At the same time, enforcement mechanisms must be developed to hold individuals accountable for unethical or illegal uses of sousveillance tools. This could involve penalties for recording without consent or sharing sensitive information without permission. Additionally, educational initiatives that raise awareness about the ethical implications of sousveillance could help foster a culture of responsible data collection and sharing, reducing the potential for harm. C. Limitations & future direction Some limitations relate to issues with real-time processing and latency to ensure seamless integration of diverse sensor inputs. Additionally, scaling ethical oversight—balancing the dual paradigms of surveillance and sousveillance—remains a significant challenge; implementing consistent ethical standards across diverse environments is critical for maintaining transparency, accountability, and fairness. Future work should therefore focus on developing more robust real-time data processing methods, adaptive feedback systems, and AI-driven sousveillance mechanisms to enhance system responsiveness and ensure ethical, privacy-preserving operations. Moreover, further user-centric studies are essential to better understand and refine the human-AI interface, ensuring that the system effectively meets user interaction needs while advancing our commitment to social good. While current object tracking technologies provide a foundation for outward-facing sensing, they face challenges when implemented in mobile, resource-constrained wearable devices. Jung’s meta-learning approach for real-time object tracking with efficient model adaptation and channel pruning offers promising directions for addressing computational limitations in sousveillance systems [23]. Future work should explore how such adaptive tracking frameworks can be integrated with inward-facing physiological sensing to create contextaware systems that maintain both efficiency and privacy. Additionally, expanding on Liu’s work in visibility status estimation could help sousveillance systems better understand complex environments while respecting the privacy of other individuals captured by outward-facing sensors [24]. Furthermore, recent studies have demonstrated that multimodal sensor fusion can significantly enhance both the robustness and privacy of integrated wearable systems, offering a promising avenue for ethically sound implementations [25]. REFERENCES [1] A. Pantelopoulos and N. G. Bourbakis, “A survey on wearable sensorbased systems for health monitoring and prognosis,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 40, no. 1, pp. 1–12, 2010. [2] S. Mann, “Surveillance (oversight), sousveillance (undersight), and metaveillance (seeing sight itself),” in 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2016, pp. 1408–1417. [3] B. C. Newell, “Introduction: The state of sousveillance,” Surveillance & Society, vol. 18, no. 2, pp. 257–261, 2020, licensed under a Creative Commons Attribution Non-Commercial No Derivatives license. [Online]. Available: https://ojs.library.queensu.ca/index.php/ surveillance-and-society/index [4] S. Mann, P. V. Do, D. E. Garcia, J. Hernandez, and H. Khokhar, “Electrical engineering design with the subconscious mind,” in 2020 IEEE International Conference on Human-Machine Systems (ICHMS). IEEE, 2020, pp. 1–6. [5] S. Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. New York: PublicAffairs, 2019. [6] K. S. Ball, “Surveillance society: Monitoring everyday life,” Information Technology & People, vol. 14, no. 4, pp. 406–419, 2001. [7] K. Crawford and R. Calo, “There is a blind spot in ai research,” Nature, vol. 538, pp. 311–313, 2016. [8] J. Thatcher, “Sousveillant media,” Understanding spatial media, p. 56, 2017. [9] H. Nam and B. Han, “Learning multi-domain convolutional neural networks for visual tracking,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4293–4302. [10] S. M. Mugo, W. Lu, M. Wood, and S. Lemieux, “Wearable microneedle dual electrochemical sensor for simultaneous ph and cortisol detection in sweat,” Electrochemical Science Advances, vol. 2, no. 1, 2022. [11] Z. Li, Y. Wang, Z. Fan, Y. Sun, Y. Sun, Y. Yang, Y. Zhang, J. Ma, Z. Wang, and Z. Zhu, “A dual-function wearable electrochemical sensor for uric acid and glucose sensing in sweat,” Biosensors, vol. 13, no. 1, p. 105, 2023, pMID: 36671938, PMCID: PMC9855683. [12] Y. Cheng, S. Feng, Q. Ning, and et al., “Dual-signal readout paper-based wearable biosensor with a 3d origami structure for multiplexed analyte detection in sweat,” Microsyst Nanoeng, vol. 9, p. 36, 2023. [13] B. Ma, K. Huang, G. Chen, Y. Tian, N. Jiang, C. Zhao, and H. Liu, “A dual-mode wearable sensor with coupled ion and pressure sensing,” Soft Sci., vol. 4, p. 8, 2024. [14] N. Jiang, G. Chen, F. Zhou, B. Ma, C. Zhao, and H. Liu, “A dual-mode wearable sensor with electrophysiological and pressure sensing for cuffless blood pressure monitoring,” J. Mater. Chem. C, vol. 12, pp. 15 915–15 923, 2024. [Online]. Available: http: //dx.doi.org/10.1039/D4TC02494J [15] X. Zeng, H. T. Deng, D. L. Wen, Y. Y. Li, L. Xu, and X. S. Zhang, “Wearable multi-functional sensing technology for healthcare smart detection,” Micromachines, vol. 13, no. 2, p. 254, 2022, pMID: 35208378, PMCID: PMC8874439. [16] H. Wang, Z. Xiang, P. Zhao, J. Wan, L. Miao, H. Guo, C. Xu, W. Zhao, M. Han, and H. Zhang, “Double-sided wearable multifunctional sensing system with anti-interference design for human–ambience interface,” ACS Nano, vol. 16, no. 9, pp. 14 679–14 692, 2022, pMID: 36044715. [17] S. Mann, J. Nolan, and B. Wellman, “Sousveillance: Inventing and using wearable computing devices for data collection in surveillance environments.” surveillance and society, vol. 1, pp. 331–355, 2002. [Online]. Available: https://api.semanticscholar.org/CorpusID:15566915 [18] S. Mann, “Veilance and reciprocal transparency: Surveillance versus sousveillance, ar glass, lifeglogging, and wearable computing,” in 2013 IEEE International Symposium on Technology and Society (ISTAS): Social Implications of Wearable Computing and Augmediated Reality in Everyday Life, 2013, pp. 1–12. [19] J.-G. Ganascia, “The ethics of the generalised sousveillance,” The—backwards, forwards and sideways||, p. 189, 2010. [20] M. Kowalski, “Between ‘sousveillance’ and applied ethics: practical approaches to oversight,” Security and Human Rights, vol. 24, no. 3-4, pp. 280 – 285, 2014. [Online]. Available: https://brill.com/view/ journals/shrs/24/3-4/article-p280 7.xml PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 56 [21] H. CHA and J. KIM, “Ethical issues concerning health data ownership,” Korean Journal of Medical Ethics, vol. 24, no. 4, pp. 423–459, 2021. [Online]. Available: https://doi.org/10.35301/ksme.2021.24.4.423 [22] K. Ross, “Watching from below: Racialized surveillance and vulnerable sousveillance,” PMLA/Publications of the Modern Language Association of America, vol. 135, no. 2, p. 299–314, 2020. [23] I. Jung, K. You, H. Noh, M. Cho, and B. Han, “Real-time object tracking via meta-learning: Efficient model adaptation and one-shot channel pruning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 11 205–11 212. [24] X. Liu, D. Lo, and C. Thuan, “Unsupervised learning based jump-diffusion process for object tracking in video surveillance,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, 7 2018, pp. 5060–5066. [Online]. Available: https://doi.org/10.24963/ijcai.2018/702 [25] A. Wood, S. Holt, and J. Marques, “Assessing the impact of transparency in organizations.” Organization Development Journal, vol. 42, no. 3, 2024. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 57 Consider, for example, a person with a seeing aid or computer vision system that helps that person see. Such a seeing aid is likely to have a camera in it. Many establishments forbid cameras, allegedly for “privacy reasons”, but establishments owe both a duty-of-care, as well as accessibility to the public. Thus such a person using such a seeing aid cannot legally be excluded. Additionally, such a person must be provided with reasonable access to services such as washroom facilities. It is therefore the responsibility of the provider to provide access to safe privacy-preserving facilities such as accessible washrooms. In our Symposium, we addressed these issues as part of our Keynote with Alan Preyra who is a partner at Bergmanis Preyra LLP in Toronto, widely regarded as a leading expert on the Occupiers’ Liability Act [32] and the Municipal Act. IX. HUMAN–INTELLIGENCE REVISITED Human–intelligence does not make AI human. It makes the psychology of sensing unavoidable. A. Transition from AR to XR As systems move from augmentation to extension, they begin to participate in identity formation rather than merely assisting action. Psyveillance reframes this shift not as a technological problem, but as a psychological one: how to design systems that extend human sensing and reflection without replacing or redefining the self. X. CONCLUSION: PSYVEILLANCE AS A NEW PSYCHOLOGICAL DISCIPLINE Psyveillance is not a finished theory, but a starting point. It names a problem that emerges when sensing becomes continuous, interpretation becomes unavoidable, and identity itself enters the feedback loop. By foregrounding reciprocal interpretation, inspectable self-models, and epistemic agency, Psyveillance offers a framework for designing intelligent systems that support, rather than overwrite, human selfunderstanding. The question is no longer whether machines can become human, but why machines are allowed to see, interpret, and remember without being seen back. It is no longer a question of “Who watches the watchers?” but, rather, “Who watches the watching.”. Thus metaveillance [18] (the sensing of sensors and the sensing of their capacity to sense) evolves to metapsyveillance, involving not only sensing of sensing, but also representation of representation. REFERENCES [1] J. C. R. Licklider, “Man–Computer Symbiosis,” IRE Transactions on Human Factors in Electronics, vol. HFE-1, no. 1, pp. 4–11, Mar. 1960, doi:10.1109/THFE2.1960.4503259. [2] S. Mann, “Can humans being machines make machines be human?,” in Proc. Int. Conf. on Cyborgs in Ethics, Law, and Art: Crossing the Border of Humanity, Medical University of Ł´ od´ z, Poland, 2021. [3] S. Mann and C. Wyckoff, “Extended Reality,” MIT 4-405, Massachusetts Institute of Technology, Cambridge, Massachusetts, 1991. Also available at http://wearcam.org/xr.htm [4] S. Mann, “23.1 Bearable Computing” in “Wearable Computing,” Chapter 23 in The Encyclopedia of Human-Computer Interaction, 2nd Ed. by Interaction Design Foundation, https://www.interactiondesign.org/literature/book/the-encyclopedia-of-human-computerinteraction-2nd-ed [5] A. Clark, Being There: Putting Brain, Body, and World Together Again. Cambridge, MA, USA: MIT Press, 1996. [6] B. Shneiderman, “Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy,” Int. J. Human–Computer Interaction, vol. 36, no. 6, pp. 495–504, 2020. [7] E. Goffman, The Presentation of Self in Everyday Life. New York, NY, USA: Overlook Press, 1959. [8] G. H. Mead, Mind, Self, and Society. Chicago, IL, USA: University of Chicago Press, 1934. [9] L. Floridi, The Fourth Revolution: How the Infosphere Is Reshaping Human Reality. Oxford, U.K.: Oxford University Press, 2014. [10] M. Merleau-Ponty, Phenomenology of Perception, C. Smith, Trans. London, U.K.: Routledge, 1962. [11] S. Mann, “Humanistic intelligence,” in Ars Electronica: Fleshfactor—Informationsmaschine Mensch, pp. 217–231, 1997. [12] S. Mann, “Mediated Reality,” M.Sc. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 1994. [13] S. Mann, “Humanistic computing: ‘WearComp’ as a new framework and application for intelligent signal processing,” Proc. IEEE, vol. 86, no. 11, pp. 2123–2151, 2002. [14] S. Mann and C. Manders, “Programming for EyeTap systems,” Embedded Linux Journal, vol. 6, pp. 43–46, Nov./Dec. 2001. [15] S. Mann, J. Nolan, and B. Wellman, “Sousveillance: Inventing and using wearable computing devices for data collection in surveillance environments,” Surveillance & Society, vol. 1, no. 3, pp. 331–355, 2003. [16] S. Mann, “Sousveillance: Inverse surveillance in multimedia imaging,” in Proc. 12th ACM Int. Conf. Multimedia, New York, NY, USA, Oct. 2004, pp. 620–627. [17] S. Mann, “Veillance and reciprocal transparency: Surveillance versus sousveillance, AR glass, lifeglogging, and wearable computing,” in Proc. IEEE Int. Symp. Technol. Soc. (ISTAS), Toronto, ON, Canada, Jun. 2013, pp. 1–12. [18] S. Mann, “Surveillance (oversight), sousveillance (undersight), and metaveillance (seeing sight itself),” in Proc. IEEE CVPR Workshops, Las Vegas, NV, USA, 2016, pp. 1408–1417. [19] S. Mann, “Big data is a big lie without little data: Humanistic intelligence as a human right,” Big Data & Society, vol. 4, no. 1, pp. 1–10, 2017. [20] S. Mann, R. Janzen, M. A. Ali, and K. Nickerson, “Declaration of veillance (Surveillance is Half-Truth),” in Proc. IEEE Games, Entertainment, and Media Conf. (GEM), Toronto, ON, Canada, 2015. [21] W. R. Ashby, An Introduction to Cybernetics. New York, NY, USA: Wiley, 1956. [22] J. Burrell, “How the machine ‘thinks’: Understanding opacity in machine learning algorithms,” Big Data & Society, vol. 3, no. 1, 2016. [23] M. Ananny and K. Crawford, “Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability,” New Media & Society, vol. 20, no. 3, pp. 973–989, 2018. [24] P. Maes, “Agents that reduce work and information overload,” Communications of the ACM, vol. 37, no. 7, pp. 31–40, 1994. [25] P. Maes, “The influence of interface agents on trust and perceived control,” in Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM, 2001. [26] R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 1997. [27] P. Pataranutaporn et al., “Influencing human–AI interaction by priming beliefs about AI,” Nature Machine Intelligence, vol. 5, no. 10, pp. 1076– 1086, 2023. [28] P. Pataranutaporn et al., “Wearable lab on body,” in Proc. IEEE EMBS, 2019, pp. 3327–3332. [29] P. P. Liang et al., “Factorized contrastive learning: Going beyond multiview redundancy,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023. [30] S. Mann and S. Hosseingholizadeh and A. S. Aksu and D. Tzanetakis and N. Kumar, “MoBorg: Mobility as Cyborg Prosthesis for ‘Being a Ball’.” In 2025 IEEE GEM Conference, 6 pages. [31] S. Mann and R. Janzen and T. Ai and S. Yasrebi, and J. Kawwa and M. A. Ali, “Toposculpting: Computational lightpainting and wearable computational photography for abakographic user interfaces” In 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE), 10 pages. [32] A. Preyra and E. Unrau, “The Law of Occupiers’ Liability”, emond PERSONAL INJURY LAW SERIES, December 2025. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 64 Social XR: EEG-Driven Visualization in VR Using Multi-Headband EEG and Unsupervised ML 1st Alexander Vicol Department of Electrical and Computer Engineering University of Toronto alexander[email protected] 2nd Shubham Panchal Department of Electrical and Computer Engineering University of Toronto [email protected] 3rd Steve Mann Department of Electrical and Computer Engineering University of Toronto [email protected] Abstract—Social VR/XR environments create a strong sense of shared presence, yet they often conceal the moment-tomoment internal states that support co-awareness in face-toface interaction. This project investigates whether lightweight, consumer-grade EEG can be transformed into an interpretable and conservatively framed shared signal for Social XR—one that supports awareness and comparison without implying clinicalgrade emotion recognition. We use Muse-class headbands and an end-to-end pipeline that combines signal processing, unsupervised learning, and Unity visualization. Our experiments use Muse S and Muse 2 headsets with four dry EEG channels (TP9, AF7, AF8, TP10) sampled at 256 Hz. From short windows, we compute baseline-normalized bandpower features and adopt a threestage modeling narrative: (i) a transparent band-ratio activation heuristic to provide an interpretable ordering reference, (ii) KMeans clustering to obtain a coarse low/medium/high partition, and (iii) a diagonal-covariance Gaussian Mixture Model (GMM) initialized from KMeans to yield smooth, soft posteriors. We map these posteriors into a continuous 0–10 score that drives Social XR visualizations (avatar color and intensity). We also introduce MuseLog, a mobile app we developed to collect raw EEG from multiple Muse headbands concurrently, enabling synchronized multi-user datasets with reduced setup friction. In addition to raw EEG, MuseLog records relative band metrics, accelerometer and gyroscope signals, and per-device battery/status; the app is also designed to accommodate optical streams (e.g., fNIRS) for newer devices such as Muse S Athena. Screenshots of the multidevice workflow and a two-person conversation pilot highlight how low-friction multi-user acquisition can support Social XR research. Index Terms—Machine learning, EEG, Social XR, VR biofeedback, unsupervised learning, KMeans, Gaussian Mixture Models, Muse S, Muse 2, multi-user data acquisition I. INTRODUCTION As VR/XR platforms shift from primarily individual experiences toward shared, persistent social spaces, an important limitation remains: many of the subtle cues that support interpersonal co-awareness are attenuated or absent. One promising route is physiological sensing, where signals such as EEG can be used to externalize relative trends in internal state that complement what avatars and voice already convey. Prior work in VR biofeedback and EEG-oriented stress or neurofeedback settings suggests that even low-density, consumergrade sensing can support meaningful state-aware interactions, so long as interpretations are conservative and grounded in protocol context [1]–[3]. However, many Social XR affect systems still assume high-density EEG caps or large labeled datasets—requirements that are often impractical for rapid prototyping. This project follows the direction highlighted in our prior Mersivity / Social XR symposium work on sharing conservative, non-clinical physiological cues to support coawareness in immersive environments [4]. In this project, we intentionally adopt a low-barrier configuration: Muse S and Muse 2 headbands, two participants, short sessions, and no frame-level affect labels. Muse-class EEG has been validated as a practical research tool in controlled paradigms, motivating its use here for course-scale prototyping [5]. Both devices offer four dry EEG channels (TP9, AF7, AF8, TP10) with reference at FPz, alongside inertial sensing (accelerometer/gyroscope) and consumer telemetry [6]–[8]. These constraints naturally motivate an unsupervised approach that prioritizes interpretability, robustness, and visualization stability over high-variance supervised classification. A practical bottleneck for scaling Social XR physiology is multi-user data collection. Many existing tools assume a single headset per recording pipeline and rely on manual synchronization across devices or applications. To address this, we built MuseLog, a mobile app that supports raw EEG capture from multiple Muse headbands simultaneously. This reduces friction in synchronized group recording and provides a cleaner substrate for future cross-user modeling, turn-taking analyses, and real-time shared feedback. II. SYSTEM DESIGN A. Hardware and data streams Our pilot uses two headbands concurrently (Muse S and Muse 2). Both provide four dry EEG channels (TP9, AF7, AF8, TP10) sampled at 256 Hz with a reference at FPz, and include an IMU for three-axis accelerometry and gyroscopy [6]–[8]. Muse 2 additionally supports PPG-based heart sensing. Newer variants such as Muse S Athena combine EEG with fNIRS for frontal oxygenation tracking, which suggests a natural path toward multimodal Social XR extensions [9]. MuseLog records, per device: •raw EEG microvolts from TP9/AF7/AF8/TP10, •absolute/relative bandpower-derived values used for fast feedback, PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 65 •gyroscope and accelerometer (IMU) channels, •headset status and battery level over time, •and is architected to log fNIRS streams when available (e.g., Muse S Athena). B. Data collection protocol and dataset scale Two users participate in a pilot study. Because the learning is unsupervised, we avoid strong affect claims and instead design the session to provide weak, ordered contextual structure that can guide interpretation. Each recording is organized into three phases: (1) a baseline segment (quiet sitting, eyes open/closed), (2) a calm condition featuring low-intensity, slow-paced XR content or conversation, and (3) a highactivation condition with more intense stimuli. The specific content matters less than the ordered timing, which provides a defensible reference for interpreting state transitions. The resulting recordings are short but sufficient for a per-user proof of concept, consistent with a project that emphasizes method clarity and appropriately bounded conclusions rather than population-level generalization. C. Preprocessing We use a compact, artifact-aware pipeline designed for consumer EEG and XR feedback constraints: •Channel selection and artifact flags: We retain TP9, AF7, AF8, and TP10. Samples are down-weighted or excluded when device-provided indicators suggest poor contact quality or artifacts. •Band-pass and downsampling: We filter to 1–40 Hz to reduce drift and high-frequency noise, then downsample by a factor of 8 to an effective rate of ≈32 Hz. This focuses the pipeline on slower state dynamics and aligns with interactive update rates in XR. •Windowing and spectral features: We segment into 0.5 s windows with 50% overlap and compute logbandpower per channel for delta (1–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (30– 40 Hz), concatenating into a fixed-length feature vector. •Baseline normalization: Features are z-normalized per user using statistics computed from baseline windows, controlling for inter-subject amplitude differences. This transforms raw EEG into a more noise-robust feature space suitable for lightweight clustering and stable visualization. D. Baseline heuristic To maintain interpretability under small-data constraints, we define a simple activation proxy: S(t) = β(t) + γ(t) α(t) + θ(t) + ϵ,(1) where ϵprevents numerical instability. This heuristic is not treated as a ground-truth label; instead, it serves as an intuitive reference for ordering the unsupervised regimes. E. Alternative 1: KMeans We apply KMeans with k= 3 on standardized feature vectors xi: min {µk},{ci}X i ∥xi−µci∥2, k ∈ {1,2,3}.(2) We select k= 3 to target an interpretable low/medium/high partition that is stable under limited data. Clusters are ordered using baseline-relative activation and the heuristic S(t)in Eq. 1, enabling a conservative mapping into calm/normal/highactivation labels. F. Alternative 2 (final): GMM refinement To better represent transitional states and reduce frameto-frame jitter, we fit a diagonal-covariance GMM initialized from KMeans: p(x) = 3 X k=1 πkN(x|µk,Σk).(3) The diagonal assumption trades expressivity for robustness, which is appropriate for short sessions and consumer EEG noise profiles. The model returns posteriors p(z=k|x)that are well suited for continuous visual feedback. G. Continuous 0–10 score We define a scalar score designed for smooth, interpretable rendering: s= 10 ·p(z=high) + 5 ·p(z=normal),(4) with remaining mass implicitly corresponding to calm. This yields a continuous trajectory that can drive intensity, brightness, or particle energy in Unity. H. Optional XR spiral mapping (extension) To remain compatible with earlier SWIM-inspired visualization drafts, we also include an analytic-signal formulation for band-limited rhythms: xa(t) = xb(t) + jH{xb(t)},(5) where H{·} is the Hilbert transform. Instantaneous amplitude and phase are: A(t) = |xa(t)|, ϕ(t) = arg(xa(t)).(6) These quantities can be mapped into phase-aware 3D trajectories for optional multi-band XR aesthetics. I. MuseLog mobile acquisition app A central contribution of this project is MuseLog, a mobile acquisition tool we developed to reduce friction in multi-user consumer EEG studies. While Muse-class devices are widely accessible, many existing pipelines assume a single headset or require manual time alignment across applications. MuseLog addresses this gap by enabling two (and extensibly more) Muse headbands to be connected and recorded concurrently from one Android device, using a unified session interface and a consistent export schema. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 66 Fig. 1. MuseLog multi-headband session UI. Two Muse devices are connected and recorded concurrently. The interface exposes per-device signal quality and battery status while displaying raw EEG traces for rapid verification. For each connected device, MuseLog logs time-aligned streams including raw EEG from TP9/AF7/AF8/TP10, absolute/relative band metrics, IMU channels (accelerometer and gyroscope), and battery/connection status [6]–[8]. The app is also designed to support newer optical modalities when available, such as EEG + fNIRS-style streams in Muse S Athena [9]. J. Unity integration For each user, we export a compact CSV containing time_sec,score_0_10,emotion_state, and optional color anchor values. A Unity C# reader interpolates between successive rows, maps discrete states to color anchors, and modulates brightness or particle intensity using the continuous score. This design keeps calm intervals visually stable and cool-toned, while higher-activation intervals appear brighter and warmer during playback. K. Social XR VR application (Unity) We implemented a two-participant Social XR visualization prototype in Unity targeting standalone Meta Quest-class headsets with optional passthrough. The scene instantiates two parallel visual stacks (Person 1 and Person 2) with banddriven effects and optional ML summary panels, enabling sideby-side comparison during shared experiences or post-session replay. L. Social XR interpretability in VR To avoid over-claiming clinical emotion detection, the VR interface presents state as relative activation rather than discrete emotional categories. Band colors remain fixed and are Fig. 2. Social XR VR prototype in Unity with two-user visualization stacks. Band-specific smoke columns and optional spiral/ML panels are shown in a passthrough-enabled scene for side-by-side social state comparison. 0 5 10 15 20 Time (s) TP9_RAW AF7_RAW AF8_RAW TP10_RAW User 1: Colorful EEG Brainwaves (Stacked Channels) TP9_RAW AF7_RAW AF8_RAW TP10_RAW Fig. 3. User 1: stacked EEG channels (TP9, AF7, AF8, TP10). Each channel is vertically offset for clarity, revealing structured fluctuations over the protocol. reinforced by an on-scene legend, while intensity and subtle motion are driven by the continuous 0–10 score and shortwindow bandpower dynamics. The two-user layout mirrors the MuseLog export structure to ensure deterministic mapping from device and participant slot to Unity objects. Fig. 2 illustrates a passthrough view with bandpower columns, optional spirals, and lightweight ML “explainability” panels that support at-a-glance comparison while preserving access to finer temporal structure. III. RESULTS A. Raw EEG dynamics Before clustering, we examined stacked raw EEG channels for each participant to verify signal plausibility and to visually inspect how electrodes (TP9, AF7, AF8, TP10) evolved over time. Fig. 3 shows a short segment for User 1, while Fig. 4 illustrates a longer recording for User 2. In both cases, the signals exhibit clear, non-saturated oscillatory structure and visible changes across phases, which supports the use of band-based feature extraction and unsupervised modeling as a meaningful (though still conservative) representation rather than a noise-driven artifact. B. Cluster coherence and ordering Across both participants, KMeans yields three regimes that can be consistently ordered using baseline-relative activation PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 67 0 10 20 30 40 50 60 Time (s) TP9_RAW AF7_RAW AF8_RAW TP10_RAW User 2: Colorful EEG Brainwaves (Stacked Channels) TP9_RAW AF7_RAW AF8_RAW TP10_RAW Fig. 4. User 2: stacked EEG channels (TP9, AF7, AF8, TP10) over the full session. The longer recording highlights how activity changes across the different phases of the protocol. TABLE I DISTRIBUTION OF INFERRED STATES AND MEAN 0–10 SCORES. User State Frames % Mean Std 1 calm 151 22.4 2.02 0.52 1 normal 380 56.5 4.66 1.05 1 high 142 21.1 8.15 0.59 2 calm 942 45.1 1.35 0.96 2 normal 373 17.8 4.87 1.19 2 high 775 37.1 8.30 1.05 together with the heuristic in Eq. 1. The GMM refinement reduces abrupt switching by providing soft posteriors, producing smoother trajectories and consequently more stable Social XR visual feedback. C. Cluster separation and score statistics To summarize coherence of the inferred regimes, Table I reports the distribution of dominant states and the corresponding 0–10 score statistics. The separation between calm, normal, and high-activation score ranges supports interpretability of the mapping under small-data constraints. D. Qualitative alignment and affective trajectories Even with a limited dataset, we observe protocol-consistent trends: baseline windows concentrate near calm/low scores, the calm condition occupies predominantly mid-range activations, and the higher-intensity condition increases the prevalence of the high-activation regime. For interpretability, we compress the three-cluster posterior into a continuous 0–10 score and render it with a color gradient (blue = calm, orange = neutral, red = high activation). Fig. 5 suggests that User 1 dips into a calmer interval before ramping up again, while User 2 exhibits stronger oscillations with pronounced low-activation periods. While not definitive, these trajectories provide weak but useful support that the inferred regimes are not arbitrary partitions of noise. E. Social XR interpretability In Unity, calm intervals appear as cooler, lower-intensity effects, while higher-activation intervals generate warmer and more energetic visuals. The combination of a discrete label and continuous score supports both coarse interpretation 0 5 10 15 20 Time (s) 0 2 4 6 8 10 Emotion Score (0 10) User 1: Color-coded emotion track (blue=calm, orange=normal, red=angry) 0 10 20 30 40 50 60 Time (s) 0 2 4 6 8 10 Emotion Score (0 10) User 2: Color-coded emotion track (blue=calm, orange=normal, red=angry) Fig. 5. Posterior-mapped affect trajectories for User 1 (top) and User 2 (bottom). Colors encode the dominant unsupervised state (blue = calm, orange = neutral, red = high activation). Fig. 6. Two-person Social XR data collection context. (Left) Conversation while wearing Muse headbands. (Right) Conversation with the MuseLog app in active use, illustrating low-friction multi-user recording in realistic social settings. and moment-to-moment fluctuation tracking without implying clinical emotion recognition. This framing aligns with conservative interpretations used in prior VR/EEG biofeedback work [1], [2]. F. Multi-user acquisition in practice MuseLog enables synchronized two-person recordings with minimal setup overhead. The app logs raw EEG, absolute/relative band metrics, IMU channels, and per-device status/battery signals, supporting both ML pipeline development and artifact-aware analysis in more naturalistic social contexts. G. Alternative designs and hyperparameter choices We considered supervised classifiers trained on segment labels, temporal deep models on raw EEG or spectrograms, and alternative cluster counts. Given only two participants and short sessions, we prioritized an interpretable unsupervised approach to reduce overfitting risk and support stable XR visualization. Empirically, k= 3 offered the most consistent and semantically separable partition for this coursescale dataset, while the KMeans-initialized diagonal GMM produced smoother posteriors suitable for continuous scoring. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 68 H. Limitations This pilot uses a small dataset and only weak protocollevel contextual cues. The “high-activation” regime should not be interpreted as a specific emotion (e.g., anger); it reflects a higher activation state under consumer EEG constraints. Motion and facial artifacts remain central challenges for headsetbased EEG, motivating future multimodal integration (e.g., heart rate and fNIRS) and larger multi-user studies [6], [9]. IV. CONCLUSION We present a lightweight Social XR pipeline that maps consumer EEG into interpretable shared visuals without clinical over-claiming. Using baseline-normalized bandpower features, we progress from a transparent activation heuristic to KMeans and a KMeans-initialized diagonal GMM, yielding stable three-state labels and smooth 0–10 scores for Unitydriven visualization. Finally, MuseLog enables low-friction synchronized multi-headband recording, supporting scalable multi-user Social XR studies. REFERENCES [1] L. Daniel-Watanabe et al., “Using a virtual reality game to train biofeedback-based regulation under stress conditions,” Psychophysiology, 2025. [2] L. Botrel et al., “The influence of time and visualization on neurofeedback in highly immersive virtual reality,” Frontiers in Human Neuroscience, 2025. [Online]. Available: https://pmc.ncbi.nlm.nih.gov/ articles/PMC11863144/ [3] S. Gerber, J. Riddle et al., “Intuitive virtual reality based frontal-midline theta neurofeedback: A feasibility study,” Frontiers in Virtual Reality, 2025. [4] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 [5] O. E. Krigolson, C. C. Williams, A. Norton, C. D. Hassall, and F. L. Colino, “Choosing MUSE: Validation of a low-cost, portable EEG system for ERP research,” Frontiers in Neuroscience, vol. 11, p. 109, 2017. [6] InteraXon Inc., “Comparing muse headbands,” Muse Support Article, 2024. [Online]. Available: https://choosemuse.my.site.com/s/ article/Comparing-Muse-Headbands [7] ——, “Muse s technical specifications,” Technical specification sheet, 2019. [Online]. Available: https://images-na.ssl-images-amazon.com/ images/I/71A9NwYDx9S.pdf [8] ——, “Science and sensor overview for muse headbands,” Company Science Page, 2024. [Online]. Available: https://choosemuse.com/pages/ science [9] ——, “Introducing muse s athena: The next evolution in cognitive fitness,” Press release, 2025. [Online]. Available: https://www.businesswire.com/news/home/20250318250704/en/ Introducing-Muse-S-Athena-The-Next-Evolution-in-Cognitive-Fitness PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 69 Early Validation of Humanistically Intelligent Exercise System Rocklen Jeong Dept. of Elec. and Comp. Eng. University of Toronto Toronto, Canada 0009-0005-5596-9016 Steve Mann Dept. of Elec. and Comp. Eng. University of Toronto Toronto, Canada 0000-0003-0363-3690 Abstract—Neurofeedback is a powerful and effective method to controlling one’s brain state, effectively providing control over one’s physical and mental being. This paper introduces early validation of a neurofeedback system grounded in humanistic intelligence to improve exercise. The results show promise for the system’s growth and expansion. This work will be fully explored in a future conference paper. Index Terms—EEG, reinforcement learning, wearable sensors, humanistic intelligence, brain-computer interface I. INTRODUCTION Neurofeedback is a powerful that provides meaningful and valuable insight into one’s mind. Control and influence over neurological states can be a powerful tool for boosting one’s physical and mental well-being. Recent studies have shown music to be a modality in which users can interact and modulate neurosignals to create a neurofeedback loop. With modern EEG systems becoming increasingly portable, such as the Muse S Athena, this paper aims to utilize a humanistically intelligent neurofeedback system to support user exercise. Through reinforcement learning, an adaptively changing music system will be explored to observe its changes and benefits in subjects performing a simple exercise. II. BACKGROUND Bhargava [1] previously demonstrated the efficacy of a humanistically intelligent neurofeedback system in supporting meditation through visual stimuli through a simulated sequential wave imprinting machine. Other researchers have found music to be an effective way to promote focus and concentration as well, particularly with the βbands of the brain. Mann [2] has also previously utilized integral kinematicsbased exercise systems in order to monitor the stability of one’s body during various exercises, including the plank. Mann determined that absement was a valid and meaningful measure to the stability of a wobble-board plank, measuring the tilt of the board as the users utilized the system. With these systems in mind, our system aims to expand on the work of Bhargava and Mann by developing an auditorystimulus-based system to promote stability and concentration during integral kinematics-based exercises. III. METHODOLOGY Data was collected using the Muse S Athena and the MuseCroc app while subjects were asked to plank for 80 seconds. The absement values of the subject were recorded at 40-second and 80-second intervals. With the recorded EEG data, a DQN was trained off the differing brain waves of subjects through four different songs and one control state. This DQN then provided a song expected to yield the highest reward for the brain state, and that song was then played. The effects of the new music were then recorded and analysed. IV. RESULTS & DISCUSSION Using a reward function built primarily on the θ/β ratio [3], high correlation coefficients were observed across multiple measures correlated with the absement. These coefficients ranged from -0.5 to -0.7, which is expected as a lower absement implies higher stability. The humanistically intelligent system showed noticeable improvement over a randomised system, with 10%+ increases in user rewards. These results show promise in such a system, particularly with a humanistically intelligent control loop involving both the computer and the human. This study focused only on planking, but expanding it to exercises of longer duration could prove beneficial. A future conference paper will present more comprehensive results and a detailed system, including the final system, methodology, and results. REFERENCES [1] A. Bhargava, K. O’Shaughnessy, and S. Mann, “A novel approach to eeg neurofeedback via reinforcement learning,” in 2020 IEEE SENSORS, 2020, pp. 1–4. [2] S. Mann, R. Janzen, M. A. Ali, P. Scourboutakos, and N. Guleria, “Integral kinematics (time-integrals of distance, energy, etc.) and integral kinesiology,” 01 2016. [3] D. van Son, W. van der Does, G. P. H. Band, and P. Putman, “EEG Theta/Beta Ratio Neurofeedback Training in Healthy Females,” Applied Psychophysiology and Biofeedback, vol. 45, no. 3, pp. 195–210, Sep. 2020. [Online]. Available: https://doi.org/10.1007/s10484-020-09472-1 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 70 DiaShoe: A Smart Shoe for Diabetic Health Monitoring with Gait and Nanoclimate Sensing 1st Xiaoming Chen Department of Mechanical and Industrial Engineering University of Toronto 2nd Mo Yang Department of Mechanical and Industrial Engineering University of Toronto 3rd Alexander Vicol Department of Electrical and Computer Engineering University of Toronto 4th Yawen Xiao Department of Electrical and Computer Engineering University of Toronto 5th Richard Zeng Department of Electrical and Computer Engineering University of Toronto 6th Wanhe Li Department of Electrical and Computer Engineering University of Toronto 7th Steve Mann Department of Electrical and Computer Engineering University of Toronto Abstract—Diabetic foot disease is a leading cause of preventable disability and amputation. Loss of sensation (neuropathy) prevents patients from perceiving injuries or hazardous inshoe conditions. We present DiaShoe, a low-cost, multimodal footwear system integrating plantar pressure, inertial measurement (IMU), temperature, humidity, and ultrasonic sensing for adaptive power management. To address the computational constraints of wearable microcontrollers, we propose a pipeline combining TabPFN for robust feature classification with knowledge distillation. A heavy ”teacher” network distills soft probabilities into a sub-kilobyte ”student” Multilayer Perceptron (MLP) capable of running on-device. Experimental results demonstrate that the system effectively monitors the ”nanoclimate” and achieves high gait classification accuracy (96% with TabPFN), while the distilled student model offers a favorable trade-off between accuracy and computational efficiency. Index Terms—Diabetic foot, smart shoe, plantar pressure, TabPFN, knowledge distillation, wearable sensors, edge AI. I. INTRODUCTION Diabetic peripheral neuropathy significantly reduces tactile and thermal sensitivity, leaving patients vulnerable to unnoticed injuries, ulcers, and infection [1], [2]. Continuous monitoring of the in-shoe environment and gait patterns is critical for prevention [3]. However, existing solutions are often bifurcated: they either focus solely on gait mechanics using expensive pressure arrays [4] or lack the computational capacity to run modern data-driven models on wearable hardware [5]. This work aligns with ongoing Mersivity / Water-HCI symposium directions on wearable sensing, edge intelligence, and human-centered health monitoring [6]. We propose DiaShoe, which accomplishes the following: 1) Multimodal Hardware: Integration of Force Sensing Resistors (FSRs), IMU, temperature/humidity sensors, and an ultrasonic rangefinder for automated weardetection and power saving. 2) Knowledge Distillation Pipeline: A method to deploy robust gait classification on constrained microcontrollers by distilling a Transformer-based Prior-Data Fitted Network (TabPFN) [7] into a lightweight MLP. 3) Validation: Demonstration of environmental tracking and gait analysis robust to sensor drift and variability. II. METHODOLOGY A. System Architecture The DiaShoe integrates sensing modules across two distinct zones managed by synchronized microcontrollers (Arduinoclass). Plantar Sensing: Three FSRs are positioned at the heel, arch (midfoot), and forefoot to capture characteristic loadtransfer patterns. An IMU captures 3-axis acceleration for gait dynamics. Environmental Sensing: Located in the anterior region to minimize gait interference, this module monitors the ”nanoclimate” (temperature and humidity). An ultrasonic distance sensor detects foot presence, enabling the system to enter a deep-sleep mode when the shoe is doffed, optimizing battery life. B. Gait Data Pipeline & Knowledge Distillation Raw IMU data is resampled to 10 Hz and segmented into 1.6 s windows (16 time steps). We extract a 42-dimensional feature vector per window (mean, std, dynamic range, spectral energy) to feed the classifier. Teacher Model (TabPFN): We utilize TabPFN, a transformer pre-trained on synthetic tabular data, which excels at small-dataset classification without extensive hyperparameter tuning [7]. Student Model (Distilled MLP): Since TabPFN is too computationally intensive for the wearable MCU, we employ knowledge distillation [8]. The TabPFN acts as a ”teacher,” generating soft probability distributions (targets) rather than hard class labels. A lightweight MLP ”student” (<1 KB) is trained to mimic these outputs. This allows the student to learn fine-grained class relationships (e.g., transitional gait phases) that are often lost in standard supervised training. C. Environmental Monitoring The system targets the ideal foot comfort range (27– 33◦C) [9]. Data is smoothed via a Savitzky–Golay filter to PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 71 Fig. 1. Performance comparison among PCA, MiniRocket, and TabPFN. TabPFN demonstrates superior robustness to sensor noise and variability compared to the baseline PCA. remove noise while preserving trends. Threshold-based alerts trigger a buzzer if conditions indicate frostbite risk (<15◦C), burn risk (>40◦C), or maceration risk (Humidity >70%). III. RESULTS AND DISCUSSION A. Gait Classification Performance We evaluated three classification approaches: Principal Component Analysis (PCA), MiniRocket, and TabPFN on the collected gait dataset (10 activity classes). As shown in Fig. 1, the PCA baseline degrades significantly under sensor drift and noise. MiniRocket offers improved robustness via convolution-free features. TabPFN achieves the highest accuracy (96%), leveraging its pre-trained priors to effectively handle inter-subject variability and limited training data. B. On-Device Distillation Efficiency To enable edge deployment, we compared the performance of student models trained with and without distillation. Fig. 2 illustrates the efficacy of the distillation process. The distilled MLP student achieves a notable accuracy improvement (approx. 73% to 78%) compared to training on hard labels alone. By mimicking the teacher’s decision boundaries, the sub-kilobyte model becomes viable for real-time, on-shoe inference without cloud connectivity. C. Nanoclimate and Safety Validation Although visual plots are omitted for brevity, validation in a controlled chamber confirmed the system’s responsiveness. Thermal Safety: The system successfully triggered alerts within minutes when exposed to simulated external heat sources (>40◦C) and freezing conditions (<10◦C), mitigating burn and frostbite risks for distinct neuropathic patients. Humidity Control: In simulated low-ventilation scenarios, relative humidity rise was tracked accurately, triggering alerts when sustaining levels >70% to prevent skin maceration [10]. Fig. 2. Effect of knowledge distillation. The distilled student models (MLP and Linear) inherit finer decision boundaries from the TabPFN teacher, achieving higher accuracy suitable for embedded deployment. Wear Detection: The ultrasonic sensor reliably distinguished wear states, successfully gating power to the high-consumption modules. IV. CONCLUSION DiaShoe demonstrates a practical path toward affordable, continuous diabetic foot monitoring. By combining multimodal sensing with a knowledge-distillation pipeline, we successfully deploy robust AI capabilities on resource-constrained hardware. The system effectively tracks gait (96% teacher accuracy, optimized for edge) and nanoclimate risks, providing a scalable solution for preventing diabetic foot complications. Future work will focus on longitudinal clinical validation and expanding the sensor array for shear force detection. REFERENCES [1] A. Ardelean, D. F. Balta, C. Neamtu, A. A. Neamtu, M. Rosu, and B. Totolici, “Personalized and predictive strategies for diabetic foot ulcer prevention and therapeutic management: potential improvements through introducing artificial intelligence and wearable technology,” Med Pharm Rep, vol. 97, no. 4, pp. 419–428, 2024. [2] Y. Yang, B. Zhao, Y. Wang et al., “Diabetic neuropathy: cuttingedge research and future directions,” Signal Transduction and Targeted Therapy, vol. 10, p. 132, 2025. [3] A. Haron, L. Li, J. Shuang et al., “In-shoe plantar temperature, normal and shear stress relationships during gait and rest periods for people living with and without diabetes,” Scientific Reports, vol. 15, p. 8804, 2025. [4] J. Chi, Q. Zhang, Z. Zhang, A. Demosthenous, and Y. Wu, “High-resolution plantar pressure insole system for enhanced lower body biomechanical analysis,” 2025 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID:279612285 [5] J. R. Verbiest, B. Bonnech` ere, W. Saeys, P. Van de Walle, S. Truijen, and P. Meyns, “Gait stride length estimation using embedded machine learning,” Sensors, vol. 23, no. 16, p. 7166, 2023. [Online]. Available: https://pmc.ncbi.nlm.nih.gov/articles/PMC10459491/ [6] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 [7] N. Hollmann, S. M¨ uller, L. Purucker, A. Krishnakumar, M. K¨ orfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter, “Accurate predictions on small data with a tabular foundation model,” Nature, vol. 637, no. 8045, pp. 319–326, 2025. [Online]. Available: https://doi.org/10.1038/s41586024-08328-6 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 72 [8] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning and Representation Learning Workshop, 2014, arXiv:1503.02531. [Online]. Available: https://doi.org/10.48550/arXiv.1503.02531 [9] Y. Shimazaki, T. Matsutani, and Y. Satsumoto, “Evaluation of thermal formation and air ventilation inside footwear during gait: The role of gait and fitting,” Applied Ergonomics, vol. 55, pp. 234–240, 2016. [Online]. Available: https://doi.org/10.1016/j.apergo.2015.11.002 [10] N. Dhandapani, K. Samuelsson, M. Sk¨ old, K. Zohrevand, and G. K. German, “Mechanical, compositional, and microstructural changes caused by human skin maceration,” Extreme Mechanics Letters, vol. 41, p. 101017, 2020. [Online]. Available: https://doi.org/10.1016/j.eml.2020.101017 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 73 Exploring EEG as a Unified Modality for Trust Assessment in Human–Robot Interaction: A Framework for Integrated Trust Representation 1st Darya Zanjanpour University of Toronto Institute for Aerospace Studies 2nd Alexander Vicol Dept. of Elec. and Comp. Eng. University of Toronto 3rd Amin Khorram Independent Researcher University of Regina 4th Eya Ibrahim Dept. of Neuroscience McGill University 5th Yaroslav Tkachenko Independent Researcher MannLab Abstract—Trust plays a central role in human–robot interaction (HRI), influencing system safety, operational effectiveness, and human decision-making. Despite its importance, trust is difficult to quantify due to its continuous, subjective, and multidimensional characteristics. Existing trust assessment approaches largely depend on questionnaires and post hoc analysis, limiting their ability to capture the dynamic interplay between cognitive reasoning and emotional response during live interaction. While task performance measures offer insight into rational decision behavior and physiological signals reflect affective and stress-related states, these indicators are commonly evaluated in isolation. This work presents a proof-of-concept framework that explores electroencephalography (EEG) as a unified modality for trust assessment in HRI. EEG offers a multidimensional representation of neural activity across multiple frequency bands and spatial regions associated with cognitive and emotional processing. By simultaneously collecting task performance metrics and EEG signals during human–robot interaction, this study investigates whether meaningful relationships can be established between behavioral performance and neural activity. The proposed approach examines the feasibility of using EEG as a single, realtime source to represent both performance-based and emotional components of trust, with potential implications for adaptive and safety-critical autonomous systems. Index Terms—Human–Robot Interaction, Trust, Multi-Agent Systems, EEG, Workload, Reaction Time, Adaptive Autonomy I. INTRODUCTION Trust is a fundamental component of human–robot interaction (HRI), shaping how humans supervise, rely on, and collaborate with autonomous systems [1], [2]. It directly influences safety, operational efficiency, and the quality of human decision-making, particularly when automation failures or uncertainty can lead to misuse or disuse [2], [3]. Despite its importance, trust remains challenging to quantify in a rigorous and systematic manner due to its continuous, subjective, and multidimensional nature [1], [3]. Rather than existing as a single observable variable, trust emerges from the interaction of cognitive reasoning and affective processes that evolve throughout human–robot engagement [1]. A central challenge in trust measurement is the strong coupling between logical and emotional aspects of decisionmaking. Observable task performance metrics, such as accuracy, completion time, error rates, and intervention frequency, primarily reflect reliance and supervisory control policies. Such behavioral indicators are commonly used in applied operator-performance settings (e.g., drone piloting) [4]. However, affective factors—including stress, uncertainty, and mental workload—also influence trust-related behavior and are not directly observable through performance alone [5], [6]. These emotional and cognitive components are often integrated in ways that make them difficult to disentangle using traditional measurement techniques [3]. When trust in automation is miscalibrated, operators may either over-rely on the system (misuse) or avoid relying on it even when appropriate (disuse). This motivates trust calibration: aligning perceived trust with the system’s actual capability [7]. Prior work also emphasizes designing for appropriate reliance and recognizing that trust evolves with experience and context [8]. At the same time, many EEG features are sensitive to workload, attention, and stress, which may confound trust inference. Future studies should therefore measure workload explicitly (e.g., NASA-TLX) and account for it as a covariate when mapping EEG features to trust-related outcomes [9], [10]. Prior work has demonstrated the feasibility of estimating trust-related states from psychophysiological signals, including EEG-based trust sensor models. These findings motivate our focus on EEG features as a practical pathway toward continuous, real-time trust monitoring [11], [12]. This work aligns with ongoing Mersivity and Water-HCI directions on wearable sensing and real-time human-centered interaction in situ [13]. Conventional trust assessment methods rely heavily on subjective questionnaires and self-reports administered after interaction has concluded [1], [3]. While useful for capturing overall perceptions, these methods suffer from recall bias and limited temporal resolution, preventing insight into how trust evolves moment-to-moment during interaction [1]. Physiological measures can provide additional information about PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 80 affective state (e.g., stress and workload) relevant to trust calibration, but are frequently treated independently from performance metrics. As a result, existing approaches often yield fragmented representations of trust rather than an integrated, operationally actionable perspective. This paper explores a proof-of-concept approach for representing both performance-based and affective dimensions of trust using a single sensing modality. Electroencephalography (EEG) is a promising candidate due to its ability to capture neural dynamics with high temporal resolution and to reflect cognitive load and operator-state changes during complex supervisory tasks. Building on this foundation, we investigate whether EEG can serve as a common representational framework linking behavioral performance metrics with underlying neural activity, enabling a unified view of trust dynamics [3]. To examine this possibility, EEG data and task performance metrics are collected concurrently during human–robot interaction. The analysis focuses on exploring relationships between performance measures (e.g., reaction time and interventions) and EEG features (e.g., bandpower and ratios) to assess the feasibility of mapping behavioral indicators of reliance/trust to neural signals. By framing trust assessment as a unified neural representation, this work aims to support future research on continuous and real-time trust monitoring. Establishing the feasibility of a single-source, real-time trust representation has important implications for safety-critical HRI applications. This is especially relevant in multi-agent and distributed autonomy settings, where one human may supervise multiple autonomous entities and trust calibration becomes both harder and more consequential [14], [15]. A unified approach may enable adaptive autonomy, timely intervention, and improved human–robot coordination, laying groundwork for more responsive and trustworthy autonomous systems [14], [16]. II. METHODOLOGY A. Method Overview A controlled simulation environment was developed to study trust-related behaviour in human–robot interaction using a simplified drone navigation task. Participants guided a virtual vehicle from a fixed start to a goal while avoiding randomly generated obstacles, presented through a 2D top-down interface to minimize task complexity and isolate decision-making behaviour as in Figure 1. Quantitative Trust Metrics (M1–M4) To capture different behavioural manifestations of trust, four complementary performance and behavioural metrics were defined [17]. M1 – Efficiency Metric: Measures deviation between the participant’s trajectory and the system-generated optimal path, capturing task-level alignment and learning-driven calibration. M2 – Obstacle Clearance Metric: Quantifies minimum distance maintained from obstacles, reflecting individual safety preferences and risk tolerance. Fig. 1: The simulation panel M3 – Mode Usage Metric: Represents the proportion of time spent in manual versus autonomous control, capturing reliance and control allocation behaviour. M4 – Geometric Alignment Metric: Assesses strategic similarity between user trajectories and autonomous plans using geometric distance measures, reflecting stable behavioural alignment patterns. Together, M1–M4 provide a multidimensional behavioural representation of trust, spanning dynamic adaptation and stable dispositional tendencies as post-interaction trust metrics. Mapping Behavioural Trust Metrics to Neural Signals While metrics M1–M4 capture trust-related behaviour, they become observable only after interaction has occurred. To enable continuous trust estimation, this work explores whether neural activity can serve as a real-time proxy for these behavioural trust dimensions. Electroencephalography (EEG) is investigated as a multidimensional signal that encodes cognitive and affective processes relevant to trust formation. As in Figure 2, the core objective is to examine whether behavioural trust components can be associated with EEGderived features, enabling trust to be inferred directly from neural activity. This relationship is conceptually expressed as f(EEG signals) = Trust Dimension where multidimensional EEG features act as an instantaneous representation of underlying trust states. This proofof-concept approach establishes the foundation for mapping behavioural trust metrics to neural signals in real time. B. Pilot protocol and alignment A pilot dataset was collected from N=6 participants (reported as S1–S6). Each participant completed a single session composed of 10 sequential task rounds, timed using stopwatch lap splits. Continuous wearable EEG was recorded throughout the session and exported as MindMonitor-style CSV logs. 2 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 81 Trust Dimensions T1, T2, ... Multi-Agent System (MAS Number of agents, number of robots, task settings, MAS factors Emergent behaviours, system-wide trust, new interaction patterns EEG Features S1, S2, ..., Sm f (EEG signals) = Trust Dimension in MAS Real-time trust estimation from brain activity Fig. 2: Conceptual pipeline illustrating the relationship between trust dimensions, EEG-derived features, and the proposed mapping from neural signals to trust dimensions for real-time trust estimation. [18] To align EEG with task structure without manual annotation, stopwatch laps were converted into exactly NR=10 contiguous round intervals per participant. Very short laps (10–15 s threshold, participant-dependent) were merged, then laps were either (i) snapped into 10 round durations when more than 10 laps remained, or (ii) split until 10 durations were obtained when fewer than 10 laps remained. Round intervals were assumed to start at t=0 in this pilot (configurable by a per-subject start offset). C. EEG preprocessing EEG samples marked unreliable by artifact flags (when present) were interpolated per channel; if flags indicated an unrealistically low fraction of good samples, masking was skipped to avoid excessive data removal. Signals were then denoised and standardized using: optional wavelet denoising, 1–40 Hz band-pass filtering, 60 Hz notch filtering, and average re-referencing across channels. D. Windowing and features Within each round, EEG was segmented into overlapping windows of length W=2 s with hop S=1 s. For each window, we compute: (i) Conventional features: channel-wise time-domain statistics (mean, std, RMS), Hjorth mobility/complexity, spectral entropy, and bandpowers for δ(1–4 Hz), θ(4–8 Hz), α (8–13 Hz), β(13–30 Hz), and γ(30–40 Hz), including relative TABLE I: Pilot dataset summary (derived from lap-time segmentation used in the pipeline). Subject Total time (mm:ss) Rounds Windows S1 07:23.49 10 428 S2 14:04.32 10 830 S3 10:27.85 10 612 S4 12:39.18 10 744 S5 12:51.56 10 757 S6 15:23.14 10 909 Total 72:49.54 60 4,280 bandpower and ratios (θ/β,α/θ). Bandpower is estimated from Welch PSD. (ii) Connectivity features: pairwise magnitude-squared coherence averaged within αand βbands for all channel pairs. (iii) ACT-inspired MP chirplet features: to capture nonstationary time–frequency structure, each window is downsampled to 128 Hz, projected to 1D via PCA-1, and approximated using greedy matching pursuit with K=6 Gaussianwindowed chirplet atoms spanning f0∈ {2,6, . . . , 40}Hz, chirp rates ˙ f∈ {−12,−6,0,6,12}Hz/s, and phases {0, π/2}. Six compact descriptors are reported: total MP energy, topatom dominance ratio, energy-weighted mean f0, energyweighted mean |˙ f|, signed ˙ fmean, and energy entropy. E. Outputs The pipeline writes three CSVs: (1) features_windows.csv (one row per EEG window), (2) features_rounds.csv (mean and std of window features per subject-round), and (3) qc_summary.csv (sampling-rate estimate and timing mismatch between EEG duration and stopwatch-derived duration). III. RESULTS A. Pilot dataset size (from code timing) Table I summarizes the dataset derived from the lap-time normalization used in the pipeline. Across all participants, the pipeline yields 4,280 overlapping EEG windows spanning 72:49.54 minutes of task time. B. Feature dimensionality (from code design) Each EEG window is represented by 90 numeric features: 72 channel-wise descriptors (18 per channel ×4 channels), 12 coherence features (6 pairs ×2 bands), and 6 MP-chirplet features. Round-level rows aggregate each window feature using mean and standard deviation, producing 180 round-level feature dimensions per subject-round. C. QC signal for temporal alignment For each participant, the pipeline logs the difference between EEG-derived duration (from sample count and estimated sampling rate) and stopwatch-derived total time. Large mismatches indicate imperfect synchronization between EEG start and stopwatch start; in that case the per-subject start_offset_s can be adjusted and the features regenerated. 3 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 82 D. EEG bandpower ratios across task rounds In addition to absolute and relative bandpower features, exploratory analysis was conducted on EEG bandpower ratios commonly associated with cognitive workload and vigilance. In particular, the theta-to-beta (θ/β)ratio and the delta-to-theta (δ/θ)ratio were computed on a per-window basis and aggregated at the round level using mean and standard deviation. The (θ/β)ratio is frequently linked to sustained attention and cognitive control demands, while (δ/θ) has been associated with low-frequency dominance related to fatigue, disengagement, or reduced alertness [19], [20]. Across participants, both ratios exhibited measurable variation across rounds, indicating sensitivity to changes in neural state over the course of the interaction. These ratio-based features complement conventional bandpower and coherence measures by providing compact indicators of relative spectral balance, supporting their inclusion in exploratory trust-relevant feature sets. IV. DISCUSSION This pilot demonstrates a practical route to a unified trust-relevant representation from EEG by (i) imposing an interaction-round structure using external timing, and (ii) extracting a feature set that spans stationary spectral power (bandpower/ratios), inter-channel coupling (coherence), and compact non-stationary time–frequency structure (MP chirplet descriptors). Even before modeling trust explicitly, the pipeline produces analysis-ready datasets at both fine time resolution (2 s windows) and round resolution (10 discrete stages), enabling future studies to relate neural state to round-wise task behavior. In addition to absolute and relative bandpower features, the pipeline computes compact EEG bandpower ratios, including the theta-to-beta (θ/β)and delta-to-theta (δ/θ)ratios, which are commonly associated with cognitive workload, attentional control, and vigilance [19], [20]. Variation in these ratios across task rounds suggests sensitivity to internal operator state changes that may co-evolve with trust-related behavior, such as increased monitoring effort, fatigue, or disengagement. Accordingly, these ratio-based features are treated as exploratory descriptors of operator state rather than direct or exclusive measures of trust. However, this proof-of-concept has clear constraints: •No direct trust ground truth yet: stopwatch-derived timing primarily captures performance/pace, which may correlate with trust but also with workload, learning, and strategy. •Timing uncertainty: without hardware-level synchronization, stopwatch and EEG start times may differ; QC flags this but cannot fully resolve sub-second alignment error. •Physiological confounds: EEG features sensitive to workload, stress, and attention may be entangled with trust-related appraisal, requiring careful experimental controls. •Pilot scale: N=6 supports feasibility assessment and pipeline validation, but not strong statistical claims about trust. Overall, the main contribution is infrastructural: a repeatable preprocessing/featurization pipeline that supports continuous trust monitoring research while remaining lightweight enough to run on wearable EEG and short HRI sessions. V. FUTURE STEPS Next steps to convert this pipeline into a full trust assessment study include: •Add trust labels: collect per-round self-report measures (e.g., brief Likert-scale trust calibration) and/or continuous trust sliders, and introduce event markers for notable autonomy successes or failures. •Manipulate trust drivers: systematically vary robot reliability, transparency, and autonomy level across rounds to disentangle trust effects from learning, adaptation, or fatigue. •Modeling and validation: train round-level and windowlevel models (e.g., mixed-effects regression and crossvalidated machine learning) to relate EEG-derived features to trust labels while controlling for task performance and timing effects. •EEG–behavioural trust mapping: investigate meaningful correlations between EEG features and trust dimensions captured through behavioural and performance metrics (M1–M4). The objective is to determine whether EEG signals can encode trust components that are otherwise observable only through post-interaction behavioural analysis, enabling these trust metrics to be mapped into a neural representation suitable for real-time estimation. •Feature relevance and interpretability: evaluate the relative contribution of conventional bandpower features, compact bandpower ratios (e.g., (θ/β)and (δ/θ)), coherence measures, and non-stationary descriptors to trustrelated predictions. Ratio-based features may offer a lowdimensional proxy for operator state changes (e.g., vigilance, monitoring effort, or disengagement) that interact with trust but are not explicitly captured by performance metrics alone. •Online inference: assess latency, stability, and temporal consistency of window-level inference (e.g., temporal smoothing or Bayesian filtering) to support real-time adaptive autonomy. •Robustness and generalization: improve artifact handling, incorporate baseline recordings, and evaluate generalization across sessions, task conditions, and increasing multi-agent supervision workloads. REFERENCES [1] K. A. Hoff and M. Bashir, “Trust in automation: Integrating empirical evidence on factors that influence trust,” Human factors, vol. 57, no. 3, pp. 407–434, 2015. [2] L. Bainbridge, “Ironies of automation,” Automatica, vol. 19, no. 6, pp. 775–779, 1983. 4 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 83 [3] E. K. Chiou and J. D. Lee, “Trusting automation: Designing for responsivity and resilience,” Human Factors, vol. 65, no. 1, pp. 137–165, 2023. [4] A. Doroftei, J. Grecu, and S. Popescu, “Quantitative assessment of drone pilot performance,” IEEE Access, vol. 12, pp. 22 200–22 215, 2024. [5] S. Schwerd and A. Schulte, “Operator state estimation to enable adaptive assistance in manned-unmanned-teaming,” Cognitive Systems Research, vol. 67, pp. 73–83, 2021. [6] S. Bhat, J. B. Lyons, C. Shi, and X. J. Yang, “Clustering trust dynamics in a human–robot sequential decision-making task,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 8815–8822, 2022. [7] R. Parasuraman and V. Riley, “Humans and automation: Use, misuse, disuse, abuse,” Human Factors, vol. 39, no. 2, pp. 230–253, 1997. [8] J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” Human Factors, vol. 46, no. 1, pp. 50–80, 2004. [9] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Human Mental Workload, P. A. Hancock and N. Meshkati, Eds. North-Holland, 1988, pp. 139–183. [10] B. Raufi and L. Longo, “An evaluation of the eeg alpha-to-theta and theta-to-alpha band ratios as indexes of mental workload,” Frontiers in Neuroinformatics, vol. 16, p. 861967, 2022. [11] W.-L. Hu, K. Akash, N. Jain, and T. N. Reid, “Real-time sensing of trust in human-machine interactions,” IFAC-PapersOnLine, vol. 49, no. 32, pp. 48–53, 2016. [12] K. Akash, W.-L. Hu, N. Jain, and T. N. Reid, “A classification model for sensing human trust in machines using eeg and gsr,” ACM Transactions on Interactive Intelligent Systems, vol. 8, no. 4, 2018. [13] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 [14] M. Chen, S. Nikolaidis, H. Soh, D. Hsu, and S. Srinivasa, “Planning with trust for human–robot collaboration,” in Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction, 2018, pp. 307– 315. [15] S. Nikolaidis, M. Chen, H. Soh, D. Hsu, and S. Srinivasa, “Trust-aware decision making for human–robot collaboration: Model learning and planning,” ACM Transactions on Human–Robot Interaction, vol. 9, no. 2, 2020. [16] T. Hagenow, K. Yen, and F. Liu, “Coordinated multi-robot shared autonomy based on scheduling and demonstrations,” IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 820–827, 2023. [17] D. Zanjanpour, H. Chen, H. H. T. Liu, and J. Plaks, “Quantitative trust and behavioral measurement tools for optimizing human–robot interaction in advanced air mobility systems,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024, accepted. [18] D. Zanjanpour and H. Liu, “Designing trustworthy human multi-agent systems for advanced air mobility through quantitative, multidimensional, real-time trust measurement,” Ph.D. dissertation, University of Toronto, 2025, phD proposal. [19] I. Marcantoni, R. Assogna, G. Del Borrello, M. Di Stefano, M. Morano, S. Romagnoli, C. Leoni, G. Bruschi, A. Sbrollini, M. Morettini, and L. Burattini, “Ratio indexes based on spectral electroencephalographic brainwaves for assessment of mental involvement: A systematic review,” Sensors, vol. 23, no. 13, p. 5968, 2023. [20] D. van Son, P. Putman, A. Angelidis et al., “Electroencephalography theta/beta ratio covaries with mind wandering and cognitive control metrics,” Annals of the New York Academy of Sciences, vol. 1454, p. 52–64, 2019. 5 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 84 Chirplet-Based Analysis of Gravitational-Waves: From Classical Chirplets to Best Chirplet Chains 1st Alexander Vicol Department of Electrical and Computer Engineering University of Toronto 2nd Syed Ali Sher Department of Electrical and Computer Engineering University of Toronto 3rd Xiaoming Chen Department of Electrical and Computer Engineering University of Toronto 4th Steve Mann Department of Electrical and Computer Engineering University of Toronto Abstract—Gravitational-wave (GW) signals from compact binary coalescences are highly non-stationary and chirp-like, with time-varying instantaneous frequency and amplitude. Standard time–frequency approaches such as the short-time Fourier transform, wavelet transform, and the Q-transform provide important tools for burst search pipelines, but their fixed time–frequency tilings can be suboptimal for rapidly evolving chirps. Chirplet transforms generalize Gabor and wavelet atoms by introducing a chirp-rate parameter and, more broadly, families of time-varying modulations. In this paper we review classical chirplet theory, connect it to the physics of GW polarizations, and illustrate how the Best Chirplet Chain (BCC) framework can be used to detect and reconstruct the GW150914 binary black hole merger in open LIGO data. We show a simple Python implementation that downloads and whitens public strain data, computes a Gaussianchirplet transform, and builds a BCC-style chirplet chain that tracks the inspiral chirp in the time–frequency plane, and briefly outline a similar workflow for GW190521. Index Terms—Chirplet, chirplet transform, best chirplet chain, gravitational waves, LIGO, GW150914, GW190521. I. INTRODUCTION Gravitational waves (GWs) are among the most fascinating phenomena in the observable universe. Their existence was predicted by the general theory of relativity, and in the weakfield regime they can be modeled as small perturbations of flat spacetime, gµν =ηµν +hµν ,|hµν | ≪ 1,(1) where ηµν is the Minkowski metric and hµν encodes the gravitational radiation. The detection of binary black-hole mergers such as GW150914 by the LIGO and Virgo collaborations marked the beginning of gravitational-wave astronomy [1]. Ground-based detectors such as LIGO are essentially giant laser interferometers that measure the fractional change in arm length, δL/L, induced by an incident GW. For astrophysical sources at hundreds of megaparsecs, the strain amplitude is of order 10−21, requiring extremely sensitive instrumentation and sophisticated signal-processing pipelines. Compact-binary coalescences (CBCs) produce chirp signals: their instantaneous frequency increases as the orbit shrinks due to gravitational radiation reaction. Matched filtering against families of template waveforms is optimal when accurate models are available, but can be fragile for sources with uncertain phase evolution. Chirplet-based methods offer a complementary approach that trades model specificity for robustness, while still specifically targeting chirp-like signals. This work also aligns with ongoing Mersivity and Water-HCI symposium directions on real-time sensing, time–frequency interpretation, and human-centered understanding of complex signals in situ [2]. II. FROM GABOR AND WAVELET TRANSFORMS TO CHIRPLETS In signal processing, the chirplet transform is obtained by correlating a signal x(t)with a parameterized family of chirplets, typically defined as chirp-modulated and shifted windows. A commonly used form is the Gaussian chirplet ψθ(t) = Aexp−(t−t0)2 2σ2exp2πi f0(t−t0) + d 2(t−t0)2, (2) where θ= (t0, f0, σ, d)collects the center time, center frequency, scale (or quality factor), and chirp rate d, and Ais a normalization constant [3]. A. Foundational Work: Mann, Haykin, Mihovilovic, and Bracewell Mann and Haykin coined the term “chirplet” and introduced the chirplet transform as a generalization of Gabor’s logon transform, with applications to radar and marine safety in dangerous waters [4], [5]. It was argued from physical considerations that many real-world signals exhibit projective or propagation-induced chirps, motivating analysis atoms that can capture time-varying frequency. Mihovilovic and Bracewell proposed an adaptive chirplet representation of signals in the time–frequency plane, foreshadowing later developments in adaptive time–frequency analysis [6]. B. Simulated Chirplet Example The following MATLAB code generates a sampled chirp and applies a Gaussian window centered on the signal: fs = 8000; % sampling frequency t = 0:1/fs:0.1; % 0.1 s duration x = chirp(t, 1000, 1, 2000);% 1 kHz -> 2 kHz linear chirp M = length(x); n = (-(M-1)/2 : (M-1)/2)’; sigma = 50; % choose window width w = exp(-n .*n ./ (2 *sigma .*sigma)); xw = w(:) .*x(:); % windowed chirp PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 85 Fig. 1: Simulated Gaussian-windowed linear chirp, taken from [7]. Fig. 2: Time–frequency representation of the same chirp, illustrating a monotonic up-chirp ridge. Taken from [7] Here, fs is the sampling frequency, xis a chirp sweeping from 1 kHz to 2 kHz, wis a Gaussian window parameterized by σ, and xw is the windowed signal suitable for subsequent chirplet-transform analysis. III. PHYSICS OF GRAVITATIONAL WAVES AND CHIRPLET INTERPRETATION In vacuum, the linearized Einstein equations reduce to a wave equation for the trace-reversed perturbation □¯ hµν = 0,¯ hµν =hµν −1 2ηµν h, (3) with h=ηαβhαβ and □the flat-space d’Alembertian. Choosing the transverse-traceless (TT) gauge isolates the two physical polarization degrees of freedom. For a plane wave propagating along the z-axis, the non-vanishing spatial components can be written as hTT ij (t, z) =   h+(t−z)h×(t−z) 0 h×(t−z)−h+(t−z) 0 0 0 0  ,(4) where h+and h×are the plus and cross polarizations. A laser interferometer with arm unit vector niresponds to the projected spatial strain. To leading order, δL(t) L=1 2ninjhTT ij (t),(5) and the detector output is a linear combination of the two polarizations, hdet(t) = F+h+(t) + F×h×(t),(6) where F+and F×are the antenna pattern functions. For a compact-binary coalescence, the strain in a given polarization can be written as a slowly modulated chirp, h+(t)≃A(t) cos ϕ(t), h×(t)≃A(t) sin ϕ(t),(7) with A(t)the slowly varying amplitude and ϕ(t)the rapidly varying phase. The instantaneous frequency and chirp rate are f(t) = 1 2π dϕ dt ,˙ f(t) = 1 2π d2ϕ dt2.(8) Over a short time interval around t0, one may Taylor-expand the phase to quadratic order, ϕ(t)≈ϕ(t0) + 2πf(t0)(t−t0) + π˙ f(t0)(t−t0)2,(9) so that the associated complex analytic signal ˜ h(t) = A(t0) exp iϕ(t)(10) locally has the same functional form as the Gaussianwindowed chirplet atom in (2). Identifying f0≃f(t0), d ≃˙ f(t0),(11) we see that the instantaneous GW frequency and its first derivative map directly onto the center frequency and chirprate parameters of the chirplet. A full GW chirp may thus be represented by a chain of such locally quadratic phase patches: a chirplet chain. IV. BEST CHIRPLET CHAINS FOR GW DETECTION A. Modeling GW Chirps Let h(t)≡h+(t) + i h×(t).(12) For a binary system observed at inclination angle ϵ, one convenient complex representation is h(t) = 1 + cos2ϵ 2A(t) cos φ(t) + icos ϵ A(t) sin φ(t).(13) In practice, the detector measures s(t)≡F+h+(t) + F×h×(t).(14) PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 86 Chassande-Mottin and collaborators introduced the Best Chirplet Chain (BCC) method to detect GW chirps without relying on a fixed bank of model waveforms [8]. The idea is to approximate an unknown chirp by concatenating chirplets whose center frequencies follow a smooth path in the time– frequency plane, subject to constraints on the continuity of the phase and its derivatives. If the phase ϕ(t)and amplitude A(t)were known, the optimal quadrature matched-filtering statistic for a template sp(tk)would be ℓp(x) =  N−1 X k=0 x(tk)s∗ p(tk) 2 .(15) Instead, one considers the generalized likelihood ratio test (GLRT), L(x) = max p∈P ℓp(x),(16) where Pdenotes the set of all allowed chirps. This infinitedimensional set is discretized into a finite family ˜ Pof chirplet chains on a grid in the time–frequency plane. B. Time–Frequency Formulation and Dynamic Programming The Wigner–Ville distribution wx(tn, fm)allows one to rewrite the quadratic statistic as a path integral in the time– frequency plane. Under reasonable approximations, ℓ˜p(x)≈ˆ ℓ˜p(x) = 1 N N−1 X n=0 wx(tn,˜ f(tn)),(17) where ˜ f(tn)is the instantaneous frequency of the chirplet chain. The problem then becomes additive over time steps, so that the maximum-likelihood chirplet chain can be found by dynamic programming (i.e., a shortest-path problem on a directed acyclic graph) [8]. Candès et al. proposed a related multiscale framework in which chirps are approximated by paths through a graph of chirplets with different time and frequency resolutions, leading to the chirplet path pursuit algorithm [9]. V. DATA AND METHODS: GW150914 EXAMPLE In this section we outline the workflow implemented in our Jupyter notebook, using open LIGO data for the first binary black-hole merger GW150914 [1], [10]. A. Data Acquisition We use the gwosc Python package to query the GW Open Science Center and download a 32 s frame of Hanford (H1) strain data around GW150914. The sampling rate is fs= 4096 Hz and the data are provided as an HDF5 file. We extract the time series h(t)and define t= 0 at the GPS time of maximum-likelihood coalescence. 864202 Time relative to event [s] 400 200 0 200 400 600 Whitened strain Whitened, filtered H1 strain around GW150914 Fig. 3: Whitened H1 strain in a 1 s window around GW150914. The binary black-hole chirp is clearly visible around t= 0. B. Preprocessing, Bandpass Filtering, and PSD Estimation We first apply a Tukey window and a high-pass / bandpass filter (e.g., 35–350 Hz) to suppress low-frequency seismic noise and high-frequency shot noise. Narrow-band spectral lines (e.g., power mains, calibration lines) are removed using frequency-domain notches. The noise power spectral density (PSD) Sn(f)is estimated via Welch averaging of off-source segments, using Hann windows and 50% overlap. We then whiten the data by dividing the Fourier-domain strain by pSn(f)and inverse-transforming back to the time domain. The resulting whitened strain ˆ h(t)has approximately unit variance and emphasizes coherent, chirp-like features. C. Gaussian-Chirplet Transform On the 1 s whitened window centered on t= 0 we compute a Gaussian-chirplet transform using atoms of the form (2). We restrict the frequency range to 40–350 Hz and the chirp-rate range to values consistent with binary inspirals in this band. The transform yields a complex coefficient C(t0, f0, d)for each chirplet; we work with its magnitude or log-magnitude. D. Toy BCC-Style Chirplet Chain Search To mimic the BCC approach, we define a coarse time– frequency grid {tn, fm}over the 1 s window and construct a directed graph whose vertices correspond to chirplet coefficients with (tn, fm)and whose edges connect temporally adjacent nodes, subject to simple slope constraints on the allowed frequency jump per time step. Each path corresponds to a chirplet chain. For each vertex (n, m)we assign a weight equal to log |Cn,m|or |Cn,m|2, and we then apply dynamic programming to find the path with maximum cumulative weight. This path provides a discrete approximation to the best chirplet chain in the sense of the BCC criterion. Finally, we reconstruct a time-domain signal by coherently summing the chirplet atoms along the best chain and compare it with the whitened strain. VI. DATA AND METHODS: GW190521 EXAMPLE In this section, we outline the process implemented for modeling another binary black-hole merger, GW190521 [11], by extending the Omega pipeline and replacing the sineGaussian wavelets with chirplets [3]. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 87 0.4 0.2 0.0 0.2 0.4 Time [s] relative to event 50 100 150 200 250 300 Frequency [Hz] Toy BCC-style chirplet-chain TF map (H1, GW150914) Best chirplet chain 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Chirplet-chain energy 1e6 Fig. 4: BCC-style chirplet-chain map for GW150914 (H1, 1 s whitened window). The color scale shows the chirplet-chain energy; the overlaid white curve indicates the highest-score chain, which tracks the inspiral chirp. A. Data Acquisition and Preprocessing To prepare the GW190521 strain data for chirplet-based time–frequency analysis, a series of pre-processing steps were applied to reduce noise and isolate the meaningful gravitational-wave signal. A 10 s segment of the strain data is extracted around the event time and a linear detrending operation is applied to remove low-frequency drifts introduced by the interferometer, shown in Figure 6. 8 6 4 2 0 2 Time relative to event [s] 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Strain (filtered) 1e 20 Band-passed & notched H1 strain around GW190521 Fig. 5: Band-passed and notched strain data used as input for the whitening and chirplet analysis of GW190521 B. Gaussian-Chirplet Transform Welch’s method is then used to estimate and visualize the noise spectrum of the detector. This data undergoes whitening, which involves taking the Fourier transform of the filtered strain data and dividing it by the square root of the power spectral density, suppressing frequencies where the noise is large and amplifying frequencies where the detector is more sensitive. The resulting plot shows the whitened strain, which appears visually more uniform in amplitude than the raw or filtered data. The power spectral density plot is shown in Fig 7 below. The filtered strain data is whitened by taking the Fourier transform, dividing by the square root of the PSD, and transforming back to the time domain. This suppresses frequencies 100101102103 Frequency [Hz] 10 25 10 24 10 23 10 22 10 21 10 20 10 19 ASD [1/ Hz] H1 strain amplitude spectral density (GW190521 segment) Fig. 6: Power Spectral Density Plot where the noise is large and amplifies frequencies where the detector is more sensitive. The resulting whitened strain appears more uniform in amplitude than the raw or bandpassed data, making the short-duration burst more apparent. This is shown in Figure 8 below. 8 6 4 2 0 2 Time relative to event [s] 500 0 500 1000 Whitened strain Whitened, filtered H1 strain around GW190521 Fig. 7: Whitened strain data for GW190521 After whitening the full 10-second segment of strain data, a program is run to extract only the 1-second interval centered on the event time (t=0). This is the time around which the gravitational-signal is expected to occur and it is the portion of the data most relevant for chirplet analysis. C. Gaussian-Chirplet Transform On the 1 s whitened window, a Gaussian-chirplet transform is computed using atoms of the form ψ(t;t0, f0, k)where t0is the time center, f0the instantaneous frequency at t0, and ka fixed chirp rate. For each time and frequency pair, a complex chirplet coefficient is evaluated: C(t0, f0) = ⟨s(t), ψ(t;t0, f0, k)⟩on the whitened strain. The magnitude of this coefficient is used to form a time-frequency map, with bright regions indicating tie and frequency locations where a chirplet with slope k matches the data. In analogy with the GW150914 example, a Gaussian-chirplet transform is computed on the whitened window and a BCC-style chirplet-chain search is performed on a coarse time–frequency grid. Because GW190521 is shorter and more burst-like than GW150914, the optimal chirplet chains tend to span fewer time samples and cover a narrower frequency band, but still show a characteristic PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 88 chirp-like structure. The time-frequency plot is shown in Figure 8. 0.4 0.3 0.2 0.1 0.0 0.1 0.2 Time [s] relative to event (t0) 50 100 150 200 250 300 350 Frequency [Hz] (f0) Simple Gaussian-chirplet TF magnitude (k = 200.0 Hz/s) 0.1 0.2 0.3 0.4 0.5 0.6 |<s, chirplet>| Fig. 8: Chirplet Template Magnitude Plot D. Chirplet Template Family Extrapolation The chirplet-template-bank code performs a simplified matched-filter search over a small family of Gaussian chirplets in order to identify which chirplet best resembles the whitened gravitational-wave data. This involves defining a parametric chirplet function, describing a Gaussian-windowed linear chirp with a time center, an instantaneous frequency, and a chirp rate. The algorithm then evaluates how well each chirple template matches the data by computing the squared inner product, |⟨x, ψ⟩|2. A small template bank is constructed using a grid of chirplet parameters, spanning several time centers, and the program exhaustively computes the matched-filter energy for every combination. The template with the maximum energy is selected as the best chirplet, representing the chirplike pattern in the data that is most consistent with this model. The parameters of the best chirplet is extracted and a timefrequency map of the template energies is plotted, illustrating where and how strongly each chirplet template matches the gravitational-wave signal: Fig. 9: Chirplet Family Model of GW E. Identifying Gravitational Wave The heat map shows which chirplet templates in the template bank best match the whitened strain data. Upon visual inspection, the brightest patch in the plot is between 50-70Hz at a time interval of 0.03 to 0.05. This means that in this short time window, the detector data most resembles a short, lowfrequency chirp. To determine if a gravitational wave actually happened at this point, it is important to first rule out the incidence of pure noise. Firstly, the loudest value in the heat map needs to be extracted: Eevent = max i,j Eij Now, we must determine if the background noise produces values this large all the time, or if it is a unique occurrence. This is done by applying the heat-map computation on data where no gravitational wave should be through a noise background. This allows for a maethematical representation of what the chirplet bank’s incrorrect detections looks like. If Eevent is typical compaerd to Eevent, the peak is likely a noise.glitch but if the event is a strong outlier, the event window may contain a chirp of interest. Quantitatively, the chirps can be compared with noise by calculating a p value: ˆp=#(Enoise≥Eevent) N After applying this procedure to the heat map, a bar chart can be obtained comparing the loudest chirplet match you get in each noise window to the loudest chirplet match in the previously found heat map. This is shown in the figure below: Fig. 10: Noise Comparison In the 10 trials conducted on the data, none of the offsource windows produced a chirplet match as loud as the event window, and a p-value of 0.0 was obtained. This means that the occurence detected by the chirplet bank in the event window is substantially louder than typical nearby noise windows. Although many more noise trials would be needed to validate this, a smaller trial size is sufficient for the purposes of this research paper. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 89 [6] D. Sato et al., “Navcog3: Large-scale blind indoor navigation with semantic features,” ACM Transactions on Accessible Computing, 2019. [Online]. Available: https://dl.acm.org/doi/10.1145/3340319 [7] Y. Zhao, E. Cutrell, C. Holz, A. Wilson, and M. R. Morris, “Seeingvr: A set of tools to make virtual reality more accessible to people with low vision,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019. [Online]. Available: https://cs.stanford.edu/∼merrie/papers/SeeingVRchi2019.pdf [8] L. M. Low et al., “Detecting and recognizing seven-segment digits using a deep learning approach,” in ITM Web of Conferences (AMICT 2023), 2024. [Online]. Available: https://www.itm-conferences.org/ articles/itmconf/pdf/2024/06/itmconf amict2023 01007.pdf [9] H. Haseeb et al., “Automated detection and recognition of sevensegment displays,” VFAST Transactions on Software Engineering, 2024. [Online]. Available: https://vfast.org/journals/index.php/VTSE/ article/download/1923/1605 [10] K. Gushima et al., “Automatic generation of seven-segment display images for reading with machine learning,” in PHM Asia Pacific (PHMAP), 2023. [Online]. Available: https://papers.phmsociety.org/ index.php/phmap/article/download/3612/2095 [11] “Imx500 intelligent vision sensor — developer world,” Sony Semiconductor Solutions, 2022. [Online]. Available: https://developer.sony.com/imx500 [12] “Realizing the world’s first image sensor with built-in ai engine (imx500),” Sony Semiconductor Solutions, 2022. [Online]. Available: https://www.sony-semicon.com/en/feature/2022101101.html [13] P. Bonazzi et al., “Tinytracker: Ultra-fast and ultra-low-power edge vision object tracking,” arXiv preprint arXiv:2307.07813, 2023. [Online]. Available: https://arxiv.org/pdf/2307.07813 [14] S. Mann, M. Condry, N. Kumar, D. Tzanetakis, C. Mann, A. S. Aksu, S. M. Lee, M. Seitz, H. Poon, R. Jeong, A. Hosseingholizadeh, H. Ossias, E. Fan, J. Ma, J. Wang, C. Uppal, N. Wood, J. P. B. Andrade, A. Chow, M. Maciesowicz, D. Zelenovic, X. Ji, M. Fan, C. Li, and T. Sharma, “27th annual mersivity / water-hci symposium,” Aug. 2025, conference proceedings (Version v1), published August 27, 2025. [Online]. Available: https://doi.org/10.5281/zenodo.16973160 PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 96 CONFERENCE CHAIR Steve Mann (PhD, MIT ’97, P. Eng., FIEEE), is widely regarded as “The Father of the Wearable Computer” [IEEE ISSCC 2000] and “father of wearable tech” (IEEE Spectrum 2025). He invented wearable computing, as well as the hydraulophone as both an acoustic instrument and as water-human-computer interaction in his childhood in the 1960s and 1970s. In the 1980s he invented HDR (high dynamic range) imaging. In 1991 Mann and Charles Wyckoff invented and coined the terms, “eXtended Reality” (XR) and “eXtended Intelligence” (XI). Mann is a Fellow of the IEEE, a founding member of the IEEE Council on eXtended Intelligence (CXI), and a tenured full professor in the Department of Electrical and Computer Engineering at the University of Toronto. He is also the recipient of the IEEE Consumer Electronics Award (2025) as well as the Lifeboat Foundation Guardian Award (2024). MASTER OF CEREMONY Ryan Janzen (PhD, UofT ’19), is a scientist, engineering researcher, and entrepreneur. Featured on the Discovery Channel, Wired magazine, and Through the Wormhole, Janzen’s innovations have been featured in 110+ international lectures, media interviews, and scientific publications. Janzen’s work has led to entirely new fields of research, including extramissive optics, veillance flux, swarm modulation, and the world’s first aircraft PLC research. His innovations have led to advances in acoustics, aerospace electronics, mathematics, and vehicle propulsion. PROCEEDINGS CHAIR Nishant Kumar (B.A.Sc, UofT 2024), is a researcher and developer specializing in brain computer interfaces (BCIs), wearable health technologies, and signal processing. He has led and contributed to multiple projects involving EEG acquisition systems, edge computing, and adaptive environments driven by physiological signals. His work spans practical engineering implementations using devices like the Muse EEG headband and ESP32 microcontrollers, often focusing on scalable, open-access solutions. He is currently an M.A.Sc. student under the supervision of Dr. Steve Mann and an IEEE student member. PUBLICATIONS CHAIR Alexander Vicol (B.Sc McMaster University), is an MEng student and researcher under Dr. Steve Mann developing extended reality (XR) and braincomputer interfaces building technologies. He has developed technologies that increase sociality with XR and BCI technology, and has led teams following Mersivity: developing technology that unites individuals with each other and nature. Papers he has co-authored include Sociality Augmentation, State-of-Float, and Gravitational Wave Detection with Chirplet Transforms. PROCEEDINGS OF THE 27th ANNUAL MERSIVITY/WATERHCI SYMPOSIUM, DEC. 4-21st 2025 97