Tangible Authoring of Embedded-Object Visualizations in Mixed Reality

Tangible Authoring of Embedded-Object Visualizations in Mixed Reality Xuyu Li School of Computer Science and Statistics, Trinity College Dublin Dublin, Ireland [email protected] John Dingliana School of Computer Science and Statistics, Trinity College Dublin Dublin, Ireland [email protected] Figure 1: An overview of our system workflow: (a)-(c) The user places a virtual heart model embedded inside a physical mannequin chest model: (a) selects the virtual representation of the chest model from the menu, and manipulates a physical box to place it, (b) selects the virtual heart model and adjusts its scaling using the slide interface, (c) finalizes the placement and stores the scene setup. (d)-(f) The user directly interacts with the real chest model to visualize its defined embedding: (d) scans the physical chest model, and the virtual heart model tracks its transform once detected, (e) selects a mode to define the cutaway outline, and uses a finger to tangibly draw a contour on the surface, (f) a corresponding cavity is generated and the virtual heart is visualized through the cavity. Abstract In this paper, we discuss work-in-progress research on using tangible interfaces for intuitively authoring and visualizing internal 3D structures in Mixed Reality (MR). Virtually embedding internal structures is an approach that is commonly used for explanatory and instructional visualizations in domains such as anatomy, engineering, and geosciences. However such embedded-object visualizations often suffer from spatial ambiguity, and can be difficult to create without technical proficiency with 3D modeling tools or graphical programming. To address these issues, we propose an approach that leverages tangible interaction in an immersive authoring system, where users curate a mixed-reality visualization from within the mixed reality experience itself. Tangible interaction, allowing the user to physically touch, hold, and feel physical elements of the MR interface, eases the modeling process and enhances the sense of the relative 3D spatial orientations and positions of objects manipulated by the user. Specifically, we provide an intuitive user Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). VINCI 2024, December 11–13, 2024, Hsinchu, Taiwan ©2024 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0967-8/24/09 https://doi.org/10.1145/3678698.3687187 interface for virtually embedding objects inside captured real-world objects, and customizing cutaways of the real object to reveal the underlying internal objects. CCS Concepts •Human-centered computing → Visualization systems and tools; Mixed / augmented reality;•Computing methodologies → Graphics systems and interfaces. Keywords Tangible Visualization, Tangible User Interfaces, Mixed and Augmented Reality ACM Reference Format: Xuyu Li and John Dingliana. 2024. Tangible Authoring of Embedded-Object Visualizations in Mixed Reality. In The 17th International Symposium on Visual Information Communication and Interaction (VINCI 2024), December 11–13, 2024, Hsinchu, Taiwan. ACM, New York, NY, USA, 5 pages. https: //doi.org/10.1145/3678698.3687187 1 Introduction Mixed Reality (MR) technologies, which blend digital information into the physical environment, have received widespread interest in the last decade. With recent developments in hardware and software technology, MR has become more reliable, cheaper, and more powerful than before. As a result of this, the demand for MR VINCI 2024, December 11–13, 2024, Hsinchu, Taiwan Xuyu Li and John Dingliana solutions has increased in mainstream application domains. One popular and recurring use of MR is for the situated visualization of interior details embedded inside real objects. Such visualizations are used for analysis or training tasks in domains such as anatomy visualization, engineering, and geosciences. Unfortunately, the development of MR applications still requires some expertise with 3D modeling or competence in graphical programming, and is not easily taken up by domain experts without training in such technologies. Furthermore, there are still a number of design challenges that need to be addressed in developing effective MR applications. Amongst these challenges is the issue of depth-ambiguity, where the relative spatial arrangement of 3D objects in the mixed reality environment is unclear or misleading. This perceptual issue is worsened in the case of embedded objects, due to the way that multiple modalities of information (real-world objects and virtual objects) essentially occupy the same space. To address this issue, such visualizations feature either a “ghosted” view, where the appearance of the outer object is modulated to look transparent, or a “cutaway” view, where a real-world object is augmented to look like a section of it has been removed to expose the interior structures. In this short paper, we present a system that allows users, without extensive technical 3D modeling experience, to intuitively generate embedded-object visualizations in MR. The approach we take is to employ tangible interface elements so that the user can physically hold, touch, and feel elements of the mixed-reality environment to address the issue of depth ambiguity and at the same time increase the intuitiveness of the interaction. 2 Related Work Tangible User Interfaces (TUIs) utilize physical objects to provide tactile feedback for digital interfaces, enhancing the user experience [ 13 , 23 ]. Users can manipulate these physical objects to directly control virtual content [ 12 ], which is found to be more intuitive and faster compared to traditional mouse and touch input paradigms used in traditional desktop 3D modeling tools [ 2 ]. The application of tangible interaction has been explored for many years such as in Virtual Reality [ 1 , 5 ], as well as Mixed and Augmented Reality [11, 20, 25, 26]. In previous works similar to ours, several studies have explored the intuitive authoring of virtual content using TUIs, such as Lee et al. [ 15 ], Muender et al. [ 21 ], and Matviienko et al. [ 19 ]. However, these tools require specific input devices or careful environmental setup in advance, which may not be affordable or available to all users. In contrast, our work aims to be accessible to the general public without the need for additional specialized devices, props, or advanced setup, utilizing only a single printed physical box to place digital assets. We allow users to directly author virtual scenes and create cutaway visualizations [ 7 , 8 ] in MR. This immersive authoring capability integrates both the experience and verification process within the same environment, following a greater WYXIWYG (What You eXperience is What You Get) view of the result and providing more natural interaction [ 14 ]. Similarly, He et al. [ 9 ] use tangible immersive authoring to create elements of the augmented reality user interface itself. In some prior cutaway works, the definitions of cutaways were procedurally defined based on the underlying objects of interest [ 4 , 16 ]. Using our system, users are able to define arbitrary shape cutaways, not only basic template geometries such as cones [ 22 ], frustums [ 18 ], and other shapes [ 24 ]. Liang et al. [ 17 ] and Coffin and Hollerer [ 6 ] works have similar objectives, but our work focuses on visualization of virtual cutaways of physical real-world objects, leveraging tangible interaction. Besançon et al. [ 3 ] previously compared tangible interaction for visualization tasks, with tactile/touch interaction, mid-air gestural interaction, and hybrid interaction, concluding it to be a promising alternative that was more natural and flexible to use. Additionally, He et al. [ 10 ] explore the use of tangible cubes for data visualization in MR, highlighting benefits such as intuitiveness, engagement, directness, etc. 3 Design and Implementation 3.1 Overview Embedded Scene Placement Physical Box Manipulation Placement Confirmation Scene Registration Cutaway Visualization Physical Object Tracking Mode Selection Cutaway Outline Definition Cavity Generation Interactive Visualization End Start Selected Model Scaling 3D Model Selection Virtual Representation of Real Object Selection True False Embedded Object Addition True False New Cutaway Generation Figure 2: Overview of our proposed system. Our proposed system is an MR prototyping tool designed for novice users to create embedded-object visualizations, through an Tangible Authoring of Embedded-Object Visualizations in Mixed Reality VINCI 2024, December 11–13, 2024, Hsinchu, Taiwan immersive authoring interface, i.e., users interact directly within the MR experience itself to create and modify objects, instead of the more typical approach of using a desktop interface such as a 3D modeling tool. Our target user group comprises domain experts from various fields who may want to quickly and intuitively generate personalized visualizations in their surroundings without needing advanced hardware setups or technical proficiency in 3D modeling or 3D programming. Specifically, our approach has three key features: • Tangible interaction. To facilitate more intuitive operation and enhance spatial understanding, tangible interaction is used throughout the process. This involves using a physical box to place 3D models, physically touching the object’s surface to create cutaways, and interacting with a menu interface projected onto the physical environment. • Immersive authoring. Authoring MR content within MR itself supports more intuitive relative positioning, allowing users to directly define virtual embedded objects and cutaway geometry based on real context. • Cutaway visualization. To improve spatial perception, we generate a 3D cavity with adjustable depth inside the physical object based on the cutaway outline definition, allowing users to view virtual inner objects. An example of the overall workflow is shown in Figure 1 and the design of our system can be divided into two parts: (1) utilizing a physical box to position, rotate and lock in place a virtual embedded object relative to a reference proxy of a real-world object, and (2) detecting and tracking the real object at run-time, and tangibly interacting with the object to create virtual cutaway geometry, through which the internal objects can be visualized interactively based on the user’s head position, as shown in Figure 2. 3.2 Embedded Scene Placement To author the mixed reality scenario, the user first selects a virtual representation of a real object from a menu. In our prototype, this real-object proxy was previously acquired using an off-the-shelf 3D scanning solution. The menu itself is projected onto a flat surface on the physical environment, leveraging the physical environment to provide tactile feedback for interactions, such as pressing buttons, thereby enhancing the robustness of interactions, compared to "midair" menus that are commonly used in mixed reality applications. Then the user manipulates a physical box, which serves as a 6-DoF controller to place the selected virtual representation (see Figure 3). By employing such a familiarly shaped object as the physical proxy, the user can intuitively manipulate and easily handle relatively complex tasks such as 3D rotation. Additionally, the physical box facilitates natural collision with the environment, simplifying the alignment of objects. Next, the user authors the embedded scene by selecting desired 3D models from the menu and placing them, again using the physical box as a widget, and can also adjust the scaling of the 3D models using a slider in the tangible user menu. The virtual representation of real objects is stylistically rendered using semi-transparent and outline effects. This allows users to position inner objects directly based on a real reference, without concern for how these transformations will be mapped in the scene Figure 3: Physical box manipulation. later. Once finished, the user can press the "confirm" button to register the current scene. The relative transform between the embedded objects and the corresponding physical object will be stored and used for the next stage. 3.3 Cutaway Visualization After defining the embedded scene associated with a physical object, the object can be detected and tracked. The corresponding embedded 3D models will automatically follow its position and orientation relative to the viewer. Using hand tracking and physical contact, users can tangibly touch the physical object’s surface and draw a closed contour to define a cutaway outline. We provide real-time visual feedback during the drawing process, as shown in Figure 4. A circular progress bar is attached to the top of the finger to represent the point confirmation progress. Once confirmation is completed, a blue point (red if it is the initial point of the path) and a red line are rendered to represent the drawing path. Figure 4: Real-time visual feedback: (a) a circular progress bar indicates that point selection is underway; (b) point selection is confirmed after a predefined dwell period. We provide users with continuous selection (see Figure 5(a)) and discrete selection (see Figure 5(b)) modes for drawing the cutaway shape to accommodate various scales of physical objects. Once the outline is completed, we generate a 3D cavity based on the cutaway’s exposed direction, used as a portal that allows users to see the virtual inner objects through it. The geometry of the VINCI 2024, December 11–13, 2024, Hsinchu, Taiwan Xuyu Li and John Dingliana cavity includes manifold surfaces that serve as an underlay mask for rendering, with walls and a floor to provide visual cues. These cues help users build a more robust mental model of the spatial arrangement of the internal and external structures. Figure 5: Selection modes. (a) The continuous selection mode. (b) The discrete selection mode. 3.4 Hardware and Software Setup Our prototype system was developed using Unity with Microsoft HoloLens 2 as the MR headset. We utilized the Mixed Reality Toolkit (MRTK) 1 for hand tracking and system configuration, and the Vuforia Engine 2 for target tracking. A physical box (8 x 6 x 3 cm, with a Vuforia-provided texture) served for the user to place 3D models during the embedded objects placement stage, and meshes for real object tracking were obtained using an off-the-shelf tool, Luma AI 3 . 4 Results Our proposed system can be used in various application domains, such as medical illustration and engineering. We demonstrate its versatility through three example scenarios: a mixed-reality anatomy visualization of a physical chest model within which is embedded an animated human heart (see Figure 6(a)); a mannequin head model inside which is embedded a virtual brain (see Figure 6(b)); and a physical desktop computer case embedded with internal hardware components, specifically a graphics card (see Figure 6(c)). Using the physical box as a tangible widget to place the 3D models is intuitive and natural for defining position and rotation. We also tested common shapes of cutaway outlines such as circles, rectangles, L-shapes, triangles, and custom shapes. The drawing can be completed in a few seconds, and the two modes provided can effectively handle objects of different scales. In addition to making the interaction more intuitive, the physical contact of the user’s hand with the real object while tracing the cutaway outline, provides a certain level of improved stability compared to midair gestures common in Mixed Reality, allowing a higher level of precision in tracing outlines. 5 Conclusions and Future Work In this work, we presented a prototype for novice users to create and visualize customized embedded scenes for real-life objects. Leveraging tactile feedback from tangible interaction and natural manipulation using a physical box, users can intuitively visualize nearby real objects without special requirements. We present its benefits and its potential for generalization in different scenarios. 1https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/ 2https://developer.vuforia.com/ 3https://lumalabs.ai/ Figure 6: Application scenarios. (a) Mannequin chest scenario. (b) Mannequin head scenario. (c) Computer case scenario. There are several potential improvements and challenges we plan to address in the development of a more comprehensive system. Firstly, we aim to enhance the embedded scene placement experience by: (1) Using a higher quality alternative to the printed physical box for a better hand-held experience. (2) Creating a more complete workflow, such as supporting the upload or download of digital assets. (3) Utilizing more robust tracking techniques to handle tracking loss caused by occlusion. Furthermore, the tangible visualization experience can be improved by: (1) Integrating more realistic rendering effects (such as capturing the real-life illumination in the scene to light the virtual objects consistently). (2) Adding surface simulation or particle effects to enhance the realism and feedback in the cutting process. (3) Incorporating audio effects that correspond to interactions, such as the sound of cutting or the impact of placing objects, to create a more immersive experience. A compelling area of future investigation is whether tangible interfaces generally facilitate improved and more engaging interaction with MR environments. To this end, we intend to conduct deeper user studies to evaluate the usability, intuitiveness and accuracy gained from our system across a larger range of scenarios. As our system does not rely on specialized input devices or advanced environment setup, it is relatively easy to deploy on mobile devices or MR headsets. We plan to extend and evaluate our approach with different output devices and also compare this against different interaction methods, such as using a Virtual Reality controller or indirect input through a touch screen. Acknowledgments This research was supported by a Trinity College Dublin PhD Award, the Science Foundation Ireland Centre for Research Training in Artificial Intelligence (Grant No. 18/CRT/6223), and the Horizon Europe Framework Program under Grant Agreement 101070109. We would like to thank Jiawen Liang for her help with the images. Test models of the Brain by Dean Lavery, Heart by Ricardo Joel Arana and Graphics Card by exéla, which appear in the rendered images, were obtained from Sketchfab.com under CC BY 4.0. Tangible Authoring of Embedded-Object Visualizations in Mixed Reality VINCI 2024, December 11–13, 2024, Hsinchu, Taiwan References [1] Jatin Arora, Aryan Saini, Nirmita Mehra, Varnit Jain, Shwetank Shrey, and Aman Parnami. 2019. VirtualBricks: Exploring a Scalable, Modular Toolkit for Enabling Physical Manipulation in VR. In Proc. 2019 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’19). ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605. 3300286 [2] Lonni Besançon, Paul Issartel, Mehdi Ammi, and Tobias Isenberg. 2017. Mouse, Tactile, and Tangible Input for 3D Manipulation. In Proceedings of the 2017 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’17). ACM, New York, NY, USA, 4727–4740. https://doi.org/10.1145/3025453.3025863 [3] Lonni Besançon, Anders Ynnerman, Daniel F. Keefe, Lingyun Yu, and Tobias Isenberg. 2021. The State of the Art of Spatial Interfaces for 3D Visualization. Computer Graphics Forum 40, 1 (2021), 293–326. https://doi.org/10.1111/cgf.14189 [4] Michael Burns and Adam Finkelstein. 2008. Adaptive cutaways for comprehensible rendering of polygonal scenes. ACM Trans. Graphics 27, 5 (Dec. 2008), 154:1–154:7. https://doi.org/10.1145/1409060.1409107 [5] Jorge Cardoso and Jorge Ribeiro. 2021. Tangible VR Book: Exploring the Design Space of Marker-Based Tangible Interfaces for Virtual Reality. Applied Sciences 11 (Feb. 2021), 1367. https://doi.org/10.3390/app11041367 [6] C. Coffin and T. Hollerer. 2006. Interactive Perspective Cut-away Views for General 3D Scenes. In 3D User Interfaces (3DUI’06). 25–28. https://doi.org/10. 1109/VR.2006.88 [7] J. Diepstraten, D. Weiskopf, and T. Ertl. 2003. Interactive Cutaway Illustrations. Computer Graphics Forum 22, 3 (2003), 523–532. https://doi.org/10.1111/14678659.t01-3-00700 [8] Steven K. Feiner and Dorée Duncan Seligmann. 1992. Cutaways and ghosting: satisfying visibility constraints in dynamic 3D illustrations. The Visual Computer 8, 5 (Sept. 1992), 292–302. https://doi.org/10.1007/BF01897116 [9] Fengming He, Xiyun Hu, Jingyu Shi, Xun Qian, Tianyi Wang, and Karthik Ramani. 2023. Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality. In Proc. 2023 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’23). ACM, New York, NY, USA, 1–14. https://doi.org/10.1145/3544548.3580704 [10] Shuqi He, Haonan Yao, Luyan Jiang, Kaiwen Li, Nan Xiang, Yue Li, Hai-Ning Liang, and Lingyun Yu. 2024. Data Cubes in Hand: A Design Space of Tangible Cubes for Visualizing 3D Spatio-Temporal Data in Mixed Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–21. https://doi.org/10.1145/ 3613904.3642740 [11] Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proc. 2016 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’16). ACM, New York, NY, USA, 1957–1967. https://doi.org/10.1145/2858036.2858134 [12] Hiroshi Ishii. 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI ’08). Association for Computing Machinery, New York, NY, USA, xv–xxv. https: //doi.org/10.1145/1347390.1347392 [13] Hiroshi Ishii. 2008. The tangible user interface and its evolution. Commun. ACM 51, 6 (June 2008), 32–36. https://doi.org/10.1145/1349026.1349034 [14] Gun A. Lee, Gerard J. Kim, and Mark Billinghurst. 2005. Immersive authoring: What You eXperience Is What You Get (WYXIWYG). Commun. ACM 48, 7 (July 2005), 76–81. https://doi.org/10.1145/1070838.1070840 [15] Jae Yeol Lee, Dong Woo Seo, and Gue Won Rhee. 2011. Tangible authoring of 3D virtual scenes in dynamic augmented reality environment. Computers in Industry 62, 1 (Jan. 2011), 107–119. https://doi.org/10.1016/j.compind.2010.07.003 [16] Wilmot Li, Lincoln Ritter, Maneesh Agrawala, Brian Curless, and David Salesin. 2007. Interactive cutaway illustrations of complex 3D models. ACM Trans. Graphics 26, 3 (July 2007), 31–es. https://doi.org/10.1145/1276377.1276416 [17] R.H. Liang, G.J. Clapworthy, M. Krokos, and R. Mayoral. 2005. Real-time predefined shape cutaway with parametric boundaries. In International Conference on Computer Graphics, Imaging and Visualization (CGIV’05). 227–231. https://doi.org/10.1109/CGIV.2005.64 [18] Endre M. Lidal, Helwig Hauser, and Ivan Viola. 2012. Design principles for cutaway visualization of geological models. In Proceedings of the 28th Spring Conference on Computer Graphics (SCCG ’12). ACM, New York, NY, USA, 47–54. https://doi.org/10.1145/2448531.2448537 [19] Andrii Matviienko, Marcel Langer, Florian Müller, Martin Schmitz, and Max Mühlhäuser. 2021. VRtangibles: Assisting Children in Creating Virtual Scenes using Tangible Objects and Touch Input. In Ext. Abstracts of the 2021 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI EA ’21). ACM, New York, NY, USA, 1–7. https: //doi.org/10.1145/3411763.3451671 [20] Kyzyl Monteiro, Ritik Vatsal, Neil Chulpongsatorn, Aman Parnami, and Ryo Suzuki. 2023. Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching. In Proc. 2023 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’23). ACM, New York, NY, USA, 1–15. https://doi.org/10.1145/3544548.3581449 [21] Thomas Muender, Anke V. Reinschluessel, Sean Drewes, Dirk Wenig, Tanja Döring, and Rainer Malaka. 2019. Does It Feel Real? Using Tangibles with Different Fidelities to Build and Explore Scenes in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi. org/10.1145/3290605.3300903 [22] Luiz Felipe Netto and Waldemar Celes. 2021. A New Focus+Context Visualization Technique for Inspecting Black Oil Reservoir Models. In 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). 25–32. https://doi.org/ 10.1109/SIBGRAPI54419.2021.00013 ISSN: 2377-5416. [23] Orit Shaer and Eva Hornecker. 2010. Tangible User Interfaces: Past, Present, and Future Directions. Foundations and Trends in Human-Computer Interaction 3, 1–2 (Jan. 2010), 1–137. https://doi.org/10.1561/1100000026 [24] Stephan Sigg, Raphael Fuchs, Robert Carnecky, and Ronald Peikert. 2012. Intelligent cutaway illustrations. In 2012 IEEE Pacific Visualization Symposium. 185–192. https://doi.org/10.1109/PacificVis.2012.6183590 ISSN: 2165-8773. [25] Aw Kien Sin and Halimah Badioze Zaman. 2009. Tangible Interaction in Learning Astronomy through Augmented Reality Book-Based Educational Tool. In Visual Informatics: Bridging Research and Practice (Lecture Notes in Computer Science), Halimah Badioze Zaman, Peter Robinson, Maria Petrou, Patrick Olivier, Heiko Schröder, and Timothy K. Shih (Eds.). Springer, Berlin, Heidelberg, 302–313. https://doi.org/10.1007/978-3-642-05036-7_29 [26] Qian Zhou, Sarah Sykes, Sidney Fels, and Kenrick Kin. 2020. Gripmarks: Using Hand Grips to Transform In-Hand Objects into Mixed Reality Input. In Proc. 2020 CHI Conf. on Hum. Fac. in Comp. Sys. (CHI ’20). ACM, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376313