How To Build a VR Headset

Alexander Albin Streißnig How To Build a VR Headset MASTER’S THESIS to achieve the university degree of Diplom-Ingenieur Master’s degree programme Computer Science submitted to Graz University of Technology Supervisor Dipl.-Ing. Dr.techn. Clemens Arth Institute of Visual Computing Advisor Dipl.-Ing. Dr.techn. Philipp Fleck Institute of Visual Computing Graz, Austria, January 2023 It’s only those who do nothing that make no mistakes, I suppose. Joseph Conrad iii Abstract Virtual Reality is now an established product used across various fields, including education, construction, and research. This work presents a modular 3D-printable VR headset that focuses on hardware customizability and software adaptability. The design includes adjustable interpupillary distance (IPD) and display-to-lens distance, which allows the headset to be configured for different users. To ensure broad compatibility, the system integrates with the OpenXR interface, runs on Windows, and supports different tracking systems. The hardware can be reproduced using 3D printing, making the headset accessible and adaptable for individual requirements. Through five iterations, each introducing certain features, the final design was developed. The headset follows a modular assembly approach, allowing individual components to be exchanged and upgraded, adding a layer of customization to the overall system. This enables the replacement of lenses, displays, and tracking hardware without redesigning the complete headset. Additionally, Monado, an open-source OpenXR runtime, was adapted with a custom driver developed for this project. The goal of this work is to provide a foundation for creating a virtual reality headset that can be further adapted and expanded for different use cases. Therefore, an instruction on how to build the headset is provided. v Kurzfassung Virtual Reality ist mittlerweile ein etabliertes Produkt, das in verschiedenen Bereichen wie Bildung, Bauwesen und Forschung eingesetzt wird. Diese Arbeit pr¨asentiert ein modulares, 3D-druckbares VR-Headset, dessen Schwerpunkt auf der Anpassbarkeit der Hardware und der Software liegt. Das Design umfasst eine einstellbare Pupillenentfernung (IPD) und einen einstellbaren Abstand zwischen Display und Linse, wodurch das Headset f¨ur verschiedene Benutzer konfiguriert werden kann. Um eine breite Kompatibilit¨at zu gew¨ahrleisten, ist das System in die OpenXR-Schnittstelle integriert, l¨auft unter Windows und unterst¨utzt verschiedene Tracking-Systeme. Die Hardware kann mittels 3D-Druck reproduziert werden, wodurch das Headset zug¨anglich und an individuelle Anforderungen anpassbar ist. In f¨unf Iterationen, in denen jeweils bestimmte Funktionen eingef¨uhrt wurden, wurde das endg¨ultige Design entwickelt. Das Headset folgt einem modularen Aufbau, sodass einzelne Komponenten ausgetauscht und aufger¨ustet werden k¨onnen, was dem Gesamtsystem eine zus¨atzliche Ebene der Anpassbarkeit verleiht. Dadurch k¨onnen Linsen, Displays und Tracking-Hardware ausgetauscht werden, ohne das gesamte Headset neu zu konstruieren. Zus¨atzlich wurde Monado, eine Open-Source-OpenXR-Laufzeitumgebung, mit einem speziell f¨ur dieses Projekt entwickelten Treiber angepasst. Das Ziel dieser Arbeit ist es, eine Grundlage f¨ur die Entwicklung eines Virtual-RealityHeadsets zu schaffen, das f¨ur verschiedene Anwendungsf¨alle weiter angepasst und erweitert werden kann. Daher wird eine Anleitung zum Bau des Headsets bereitgestellt. vii Affidavit I declare that I have authored this thesis independently, that I have not used other than the declared sources/resources, and that I have explicitly indicated all material which has been quoted either literally or by content from the sources used. The text document uploaded to TUGRAZonline is identical to the present master’s thesis dissertation. Date Signature List of Figures 3.1 Varjo XR-4 Secure Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2 On the right, we can see an eye that can keep an object in focus. This is achieved by setting the focal point on the retina of the eye. On the left eye, a strong deviation of the targeted focal point is illustrated. This strong difference in diopter is called anisometropia. . . . . . . . . . . . . . . . . . 10 4.1 Images of the individual headset prototypes starting from Iteration 1 to the finaldesign..................................... 15 4.2 In this left image, the first prototype of the display mount is shown. The idea was to mount the display using a slight press fit into the cardboard. While this was not the ideal solution, it helped generate future ideas and strategies for mounting. The right image shows the mounting of the lens holder, indicated by the white arrow. The cardboard strap can be pushed or pulled to adjust the distance between the display and the eyes. . . . . . . 16 4.3 (a) To secure the displays in place, separate mounting brackets were glued onto the cardboard. The mounting technique used a simple slot-in mechanism, where the display was inserted from the top and held in place by the connection cable. (b) On the back, another bracket was glued to hold the breakout board for each display. (c) Both mounts were attached to the backside flap of the Google Cardboard, which can be opened. (d) Closed sideoftheheadset. ............................... 17 4.4 (a) The display holders are designed to hold the Fresnel lenses in place without restriction. (b) To determine the focal length of the lenses, the mounting point for the displays can be adjusted, moving back and forth. . . 19 xvii xviii LIST OF FIGURES 4.5 (a) The lenses are now fully encased and have a fixed seat. This was done by adding a top cover to support the upper bracket of the lens holder. (b) A holder for the driver board was introduced on the bottom side of the HMD. This allowed the headset to mount all the needed electronics. . . . . 20 4.6 In (a), we can see that the top of the HMD now has screws on top where the distance between the display and lens can be adjusted. (b) The backside has also changed with a detachable slot where the driver board can be inserted . 21 4.7 (a) The front of the device houses the two separate eye modules. The top provides the mechanism for the adjustable IPD system. (b) On the back side, the electronics compartment and the camera are mounted . . . . . . . 22 4.8 (a)To secure the display, we retained the plate-mounting bracket from the previous iteration. (b) The breakout board is now fastened with screws rather than a separate mounting bracket. (c) The lens-holder system remains largely unchanged, but the top and bottom covers are now fused with the lens holder, and a dovetail slide was also added for the display plate. (d) Once assembled with the backplate, the complete inner assembly appears asshownintheimage............................... 23 4.9 (a) The module consists of two main parts: the outer component seen on the left side and the inner component on the right. The inner component can still be pushed slightly inward to complete the assembly. In Image (b), the parts are fully combined, with the lid placed on top to protect the chamber from incoming light. (c) The final component is the driver board on the back. The board is mounted backwards, as the cable exits in that direction. ..................................... 24 4.10 (a) On the top side of the case, the complete IPD adjustment system is visible, including the rack-and-pinion mechanism. The gear rack is directly connected to the lens module, allowing it to slide for adjustment. To ensure it follows the intended linear path, raised guide features are integrated into the case. Without these, the rack would deviate from its path during movement.(b) On the bottom side, the HDMI and power ports are accessible through dedicated openings. Additionally, two slots are cut out to provide access to the adjustment wheel. To ensure the device stands securely on flat surfaces, small feet have been added to the bottom of the case. . . . . 25 5.1 Software structure of the initial prototype. This describes the path of odometry data from its generation at the RGBD device to its processing and rendering at the stereo camera system. The different colors describe different Software component types. At the bottom of the image, the meaning of the individual colors are displayed. . . . . . . . . . . . . . . . . . . . . . . . . . 31 5.2 Cameras used for evaluation and comparison. . . . . . . . . . . . . . . . . . 32 LIST OF FIGURES xix 5.3 (a) The two cameras are displayed in the Unity viewport. This visualization makes the program debugging easier because the headset’s movement is visible. (b) The image shows the script parameters. There is one parameter for each camera to let the script know which cameras are being used. One parameter is for the inter-eye distance, which refers to the relative distance between the two cameras. The last two checkboxes function as buttons that can be clicked to reset the position and rotation. . . . . . . . . . . . . . . . 36 5.4 The final rendered view of the left and the right. . . . . . . . . . . . . . . . 36 5.5 The diagram shows a more advanced solution consisting of multiple additions. With the introduction of an IMU, which is included in the camera, an algorithm combining the Infrared camera and the inertia and gyroscope values is needed. This is done through a Visual-Inertial SLAM system. The relative position changes get processed and sent to the application through the OpenXR standard. The OpenXR loader passes the data forward to the individual applications. An OpenXR Application can be any application that uses the OpenXR specification to communicate with the headset. From there, the applications rendered images get rendered onto a render target, which is provided by the OpenXR loader through the OpenXR runtime, where the driver is located. These render targets can then be displayed. . . 37 5.6 In this image, we can see the Unity project. It consists of multiple objects that are necessary for the OpenXR integration and platform, with cubes on top to gain a sense of space when using it. . . . . . . . . . . . . . . . . . . . 48 5.7 This window shows the rendered scene for each eye. . . . . . . . . . . . . . 49 6.1 3D printed Parts for the eye module with an enumeration . . . . . . . . . . 61 6.2 3D printed Parts for the case and the Electronics Compartment . . . . . . 62 6.3 This image shows the monado UI consisting of multiple windows. . . . . . . 64 List of Tables 3.1 Overview of the hardware and software requirements for the project. These requirements define the functional goals of the system. . . . . . . . . . . . . 12 4.1 The learnings over each iteration, which were incorporated into the last design........................................ 27 4.2 A comparison of existing headsets and our solution. The one thing to keep in mind is that the panels and lenses are swappable, this means some metrics are dependent on the configuration. The best values are highlighted in bold, while values that can be changed through swapping of components areunderlined. .................................. 28 5.1 Specifications of investiagted cameras . . . . . . . . . . . . . . . . . . . . . 32 6.1 Components which need to be bought with their corresponding Product ID andvendor .................................... 60 6.2 Parts for one Eye module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 6.3 Parts for one Eye module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.4 This Table lists the required dependencies for Monado along with a brief description of their general purpose. These dependencies must be installed to allow CMake to generate the necessary build files successfully. . . . . . . 63 6.5 Environment variables for starting monado properly. The first one is for instantiating the headset. Second for the streaming index of the camera, third and fourth for the slam configuration and the last one to enable the debuggui. .................................... 65 xxi 1 Introduction In today’s world, virtual reality (VR) has established itself in multiple fields. An example is architecture. Architects can now create customized 3D experiences to show their clients what their future homes will look and feel like. This leads to a better decision-making process and helps avoid costly changes later on. In the industry, VR can be used to train workers through interactive courses. This is especially valuable for employees who operate in unsafe environments, as they can learn proper workflows without being exposed to real danger. The same applies to the military. Soldiers can be placed in realistic training scenarios to improve decision-making under stress. Flight simulations help pilots practice maneuvering skills without risk. Mistakes can be made and learned without the risk of consequences. In the medical field, students can interact with the human body. They can explore every muscle and organ from all angles, which helps them to improve their understanding. Seeing the body in its actual size also provides a better sense of scale, something that is difficult to grasp in textbooks [1]. As seen in these examples, VR has a broad user base with different needs and expectations. Unfortunately, consumer-grade headsets often do not offer the flexibility required for specific use cases. Ecosystems like those of the Oculus headsets are closed source and do not allow any modification of their codebase or access to internal drivers. In addition, these devices are fully sealed and not designed for disassembly, making it difficult or even impossible to replace or upgrade components. Whether the goal is to swap out the optical system or access deeper software layers to enable driver-level functionality, the limiting factor remains the same: Commercially available headsets are typically locked into fixed hardware and software configurations. Although open standards like OpenXR simplify software development for VR applications, open and modular VR hardware remains rare. An open system could improve research in multiple fields by providing a foundation on which tracking systems can be freely swapped and integrated. In industrial environments, custom VR headsets could be 3D printed and adapted to fit the needs of niche scenarios. 1 2 Chapter 1. Introduction Education could benefit from custom headset optics tailored to each individual while saving costs by reusing other shared components of the headset. The goal of this thesis is to design and implement a foundation for a modular VR device. The system should provide enough flexibility in its hardware to accommodate a wide range of use cases. The headset is designed to follow a modular approach, allowing components to be easily replaced, upgraded, or adjusted as needed. Additionally, the software solution should be fully open-source and available for modification, enabling developers and researchers to adapt it to their needs. The development process is divided into two main parts: hardware and software. The hardware is developed iteratively, starting with a non-functional cardboard prototype, and evolves into a fully 3D-printed, screw-assembled version featuring two adjustment mechanisms. On the software side, an initial prototype is created to test the feasibility of a custom implementation. Later on, an OpenXR framework is introduced to broaden the range of use and also to be compatible with already existing software. The main contributions of this work are summarized as follows: •Designing and iteratively developing a customizable, 3D printable VR headset with modular and adjustable components •Developing a new device driver for the Monado project to enable custom headset integration through OpenXR •Creating a Windows-compatible version of the tracking system for Monado using RealSense and a SLAM backend •Providing a guide on how to compile the custom driver into Monado With these contributions, the basis for a working custom VR system is set. This enables other people to work on an existing solution and not on a headset from the ground up. The following chapters provide a full overview of the project. Chapter 2 discusses related work on VR hardware, tracking systems, and open standards. Chapter 3 explains the iterative development process of the hardware, including a reflection on the advantages and limitations. Chapter 4 focuses on software integration, starting from a naive initial approach and leading to a fully functional OpenXR-compatible system using Monado, followed by a discussion. Finally, Chapter 5 summarizes the complete system and outlines possible future improvements. 2 Related Work Contents 2.1 Customisability of VR . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Standardization and Privacy . . . . . . . . . . . . . . . . . . . . . 5 2.3 Odometry and Tracking . . . . . . . . . . . . . . . . . . . . . . . 6 In this section, related projects and technologies in the field of Virtual Reality are discussed. This includes the use of standardization for cross-application compatibility, concerns regarding the handling of private data, the customizability of headsets, and tracking technologies in Virtual Reality (VR) headsets. The use of VR has increased significantly over the past few years. However, research into this technology dates back to 1968. Ivan Sutherland developed the first head-mounted display (HMD), called The Sword of Damocles [2]. The localization system had to be suspended from the ceiling due to the weight of the entire setup. Using an ultrasonic localization system, the software was able to track head movement and adjust the visuals displayed on the headset accordingly. Later, in 1989, Jaron Lanier, often referred to as ”the father of VR”, coined the term Virtual Reality[3]. He worked at VPL Research and helped develop the VPL Eyephone, a PC-powered VR headset, along with a data glove that enabled gesture-based input. In 1992, Thomas P. Caudell and David W. Mizell published a paper describing how a see-through headset could assist workers during assembly tasks [4]. In that paper, they introduced the term Augmented Reality to describe the concept. Today, both Virtual Reality and Augmented Reality fall under the broader term of technologies referred to as Extended Reality (XR), with a fluid transition between both technologies. In 1995, Nintendo, a major player in the gaming console industry [5], attempted to release its first VR headset, the Nintendo Virtual Boy. Unfortunately, the technology was not advanced or attractive enough to be a commercial success. In the work of Boyer [6], the failure of the product is not described in binary terms of success or failure, but 3 10 Chapter 3. Motivation and Requirements 3.1 Motivation Throughout the years, numerous adaptations of VR headsets have been developed. Creating various devices for different types of problem. Some devices, like the Oculus Quest series, are targeted at a broad consumer base as a budget-friendly, beginner-friendly headset. Some others, such as the Varjo XR-4 Secure Edition[39] seen in Figure 3.1 , aim to provide high comfort, high usability, and high resolution for a specialized audience in fields such as medicine or architecture. These devices cost up to e26.990,00[40], which is not in the price range of a typical consumer. However, this is still a standard headset intended for individuals within a certain range of the norm. A company that wants to sell customizable VR experiences is BigScreen VR[41], which takes a 3D Face Scan and uses it to create a customized fit around the user’s head. This results in a perfect placement of the headset, which improves the overall experience. 3.1.1 Specialised Eye condisions Certain conditions can prevent a person from perceiving stereoscopic vision. An example is monocular vision, where only one eye is functional, making stereo depth perception impossible. Another condition is anisometropia, characterized by a significant diopter difference between the two eyes. An illustration of the condition can be seen in Figure 3.2. Without specialized corrective lenses, users with such conditions cannot experience VR in the same way as the average user. However, with a customizable headset, it becomes possible to accommodate any lens or lens system. For example, a custom lens could be inserted to correct the difference in diopters, allowing the user to have a better VR experience in general. Figure 3.2: On the right, we can see an eye that can keep an object in focus. This is achieved by setting the focal point on the retina of the eye. On the left eye, a strong deviation of the targeted focal point is illustrated. This strong difference in diopter is called anisometropia. 3.1. Motivation 11 Another use case is accommodating users with non-normative head or eye geometries. For example, individuals with hypertelorism, an abnormally large distance between the eyes, may struggle to find a compatible VR headset, if any exists at all. If the IPD system cannot match the user’s requirements, potential side effects include double vision, eye strain, or a complete inability to focus on the virtual image. With a modular system, the IPD can be individually adjusted, allowing customized alignment of the lenses and displays to meet the user’s specific needs. This ensures a functional and comfortable VR experience even for users with unusual anatomical features. 3.1.2 Modular Design for Testing and Evaluation The field of optics and displays is constantly evolving. Twenty years ago, some of the most popular display technologies were Liquid Crystal Displays (LCDs) and Plasma TVs. At the time, these were considered state-of-the-art and were widely used. Today, with the rise of OLED, QLED, Micro-LED, and other advanced technologies, these older panel types feel significantly outdated. VR lens systems have also evolved, starting from as simple aspherical lenses and progressing to more modern designs such as pancake lenses. Pancake lenses are lighter than aspherical lenses and offer better visual quality compared to Fresnel lenses. In addition, different tracking systems have their unique characteristics. For example, an optical RGB tracking system will not function correctly if there is insufficient lighting in the scene. When tracking with an inertial measurement unit, drift can occur over time, resulting in a position that does not accurately reflect the location of the real-world. Therefore, different scenarios require different technologies. Dark scenes, for example, can be tracked using an infrared camera in combination with a random pattern projector. These changing requirements create the need for flexible testing solutions. To evaluate new display technologies, with or without specific optical paths, new hardware must be designed, tested, and validated. This results in redundant work hours, which can be optimized. With a testbed that already incorporates essential features, the development time can be reduced, allowing research into new technologies to progress more efficiently. 3.1.3 Shareable Components As mentioned in the previous section, components can be altered and reused. When using a custom-fitted headset, both hardware and software should be able to adapt without requiring hours of adjustment. This would not only interrupt the user’s experience, but also create additional obstacles for a more inclusive VR setup. An easy replacement mechanism would enable the reuse of components, which also helps reduce e-waste, as individual parts can be sold or replaced separately. Furthermore, by enabling quick swapping of components, multiple users, such as those in a school setting, can share certain hardware when it is not needed simultaneously. In this way, some parts 12 Chapter 3. Motivation and Requirements can be specialized and custom-made for each person, while shared components help lower the acquisition cost per user. This modular approach also enables for easier device repair. If some component fails, the replacement can be mounted without the need for specialized equipment. This can also be used to upgrade certain components without the need to discard the entire system. 3.1.4 Standardised Interface A user browsing the internet relies on many standardized interfaces. These interfaces enable cross-application compatibility without requiring specialized implementations. Examples include the Universal Serial Bus (USB), the Transmission Control Protocol/Internet Protocol (TCP/IP), and the Compute Unified Device Architecture Runtime Application Programming Interface (CUDA Runtime API). In the XR world, the OpenXR interface provides a similar function by enabling seamless communication between VR applications and hardware. Without such an interface, VR or AR applications developers would need to implement specialized code for each headset, resulting in significantly more work and higher development costs. It would also create a barrier for new headsets entering the market, as established devices would have a strong advantage due to developers already being familiar with their communication protocols. 3.2 Requirements Based on the motivation discussed previously, a set of requirements was derived (see Table 3.1). These requirements aim to ensure that, if fully met, the project reaches a certain level of usability. A major key point is the modular assembly of the system. It begins with a module that houses both the display and the lens (R1). This module should be designed to allow Requirement Description R1 Replaceable lens system and display mounting R2 Detachable compartment for optical tracking and miscellaneous electronics R3 Displays are sealed against incoming light R4 Separate adjustment system for inter-pupil distance and lens to display depth R5 Usable in Windows to enable a broader user base R6 Use a standardised interface for broader application compatibility R7 Visual-Inertia Tracking for exemplary integration of a head tracking system in Windows Table 3.1: Overview of the hardware and software requirements for the project. These requirements define the functional goals of the system. 3.2. Requirements 13 for easy attachment and removal without requiring complete disassembly. Another requirement, also falling into the category of modular systems, is the detachable compartment for the camera electronics and the camera mount (R2). As in the previous requirement, this compartment should be removable without major disassembly. The core concept of the headset is customizability, allowing users to adjust it to their preferences. An adjustable IPD mechanism is necessary, as this parameter varies between users. Similarly, when using different lenses, the display-to-lens distance must be adjusted to account for varying focal lengths (R4). Another key challenge is light leakage, which reduces immersion. To address this, the headset should be designed to minimize external light intrusion (R3). However, the system does not end with hardware alone. Integration of software is also necessary for the VR system to work. To ensure compatibility with a wide range of applications, OpenXR support must be included (R6). This allows the headset to work even with applications that are not specifically designed for it. Furthermore, making the application accessible on Windows increases the potential user base (R5). Finally, a VR headset would not be functional without some form of positional tracking. Therefore, the system must include a way to adapt to various tracking methods (R7). As a final summary, the goal of this project is to construct a modular VR headset with adjustable IPD and display-to-lens distance, as well as software integration that supports the OpenXR interface, is compatible with Windows and can adapt to different types of tracking systems. 4 Iterative Hardware Design Contents 4.1 Cardboard - Iteration 1 . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2 Google Cardboard - Iteration 2 . . . . . . . . . . . . . . . . . . . 17 4.3 Introduction of 3D Printing - Iteration 3 . . . . . . . . . . . . . 18 4.4 Designing a Case - Iteration 4 . . . . . . . . . . . . . . . . . . . . 19 4.5 Stability Improvement - Iteration 5 . . . . . . . . . . . . . . . . 20 4.6 FinalDesign .............................. 21 4.7 Discussion of Hardware . . . . . . . . . . . . . . . . . . . . . . . . 26 Iteration 1 Iteration 2 Iteration 3 Final DesignIteration 5Iteration 4 Figure 4.1: Images of the individual headset prototypes starting from Iteration 1 to the final design. 15 16 Chapter 4. Iterative Hardware Design This section documents the process of designing the headset. Using rapid prototyping as the foundational approach, the headset underwent many cycles of redesign and reevaluation, starting from a cardboard-only prototype without any electronics and evolving into a fully 3D-printed modular system. With each iteration, the additions and insights were documented and used to generate a requirement list for the next design cycle. Throughout each design cycle, new requirements were added to progressively move closer to the desired specification goals. With this approach, new ideas can flow more easily into the headset’s system, while also enabling controlled direction and effective milestone tracking. 4.1 Cardboard - Iteration 1 The goal of the first prototype was to gain insight into the challenges and obstacles of designing a head-mounted display. Inspiration was drawn from the Oculus Rift [31] and Google Cardboard [42] to start the process. The concept involved designing a mounting bracket for the displays, allowing them to move back and forth (see Figure 4.2), and incorporating a Google Cardboard lens. Ultimately, the prototype was not fully developed because the effort required exceeded the benefits. The conclusion was to utilize an existing solution, such as the Google Cardboard HMD, and to design mounting brackets to secure the displays behind the lenses. Two insights were concluded from this process. First, mounting the displays and lens system proved to be more complex than expected, especially since the goal was to have (a) display holder (b) displayed holder and mounting bracket Figure 4.2: In this left image, the first prototype of the display mount is shown. The idea was to mount the display using a slight press fit into the cardboard. While this was not the ideal solution, it helped generate future ideas and strategies for mounting. The right image shows the mounting of the lens holder, indicated by the white arrow. The cardboard strap can be pushed or pulled to adjust the distance between the display and the eyes. 4.2. Google Cardboard - Iteration 2 17 a replaceable lens system. Second, designing an adjustable inter-eye distance mechanism requires careful use of space to avoid obstructing the user’s view. Creating a cardboard prototype is a feasible step for identifying potential obstacles early in development, which can lead to fewer iteration cycles, even if the cardboard solution is not ultimately successful. Learning 1 4.2 Google Cardboard - Iteration 2 As a second iteration, Google Cardboard was used as a base, with the idea of replacing the smartphone holder with two separate displays. This led to the creation of the first mounting brackets for the displays and breakout boards, as shown in Figure 4.3. This (a) display holder (b) mounting bracket for breakoutboard (c) cardboard cover open (d) cardboard cover closed Figure 4.3: (a) To secure the displays in place, separate mounting brackets were glued onto the cardboard. The mounting technique used a simple slot-in mechanism, where the display was inserted from the top and held in place by the connection cable. (b) On the back, another bracket was glued to hold the breakout board for each display. (c) Both mounts were attached to the backside flap of the Google Cardboard, which can be opened. (d) Closed side of the headset . 18 Chapter 4. Iterative Hardware Design iteration also introduced 3D printing into the project, which, due to its accessibility and speed, enabled faster prototyping and improved the design process. Since the displays lacked mounting holes, they were slotted into the holders from the top and secured at the bottom by the connection cables. The breakout board brackets were designed similarly, with the boards held in place by their connection cables. For each side of the headset, a display holder and a breakout board holder were required, both of which were glued to the surface of the cardboard headset. The prototype was tested with two pre-rendered images, which produced a stereoscopic effect when looking through the lenses. Mounting of the display and breakout board with the current holder proved to be a feasible solution and could be implemented in future iterations. However, the tolerance of the holder was too loose, resulting in small positional deviations of up to 2 millimeters, which is critical for this immersive application. In addition, the displays tended to pop out of their sockets due to the loose fit. The driver board was still left hanging freely because the Google Cardboard did not have sufficient space for it. Furthermore, the headset cardboard hinge was slightly damaged from repeated opening and closing of the flap, leading to the conclusion that cardboard would not be suitable as a base material for future prototypes. The displays should not move and be well seated or screwed, as this breaks user immersion and causes discomfort when wearing. Learning 2 4.3 Introduction of 3D Printing - Iteration 3 To increase the stability and durability of the headset, plastic was used in this iteration, making it the first 3D-printed headset in the design process. For development, PLA was chosen as the material, as it is one of the most widely used, reliable, and fastest materials for 3D printing [43]. Most of the design work was done in a CAD program called Fusion 360 [44], where the entire design process was digitalized. The model files were then transferred to a slicer, which divides the model into individual layers and calculates the path of the printhead. This prototype, visible in Figure 4.4, also introduced Fresnel lenses, with the challenge of determining their focal length. A system was designed that allows for manual adjustment of the distance between the lens and the display. A lens bracket was created to hold the Fresnel lens and a floor plate was used to secure the brackets. The displays were mounted on a sliding plate that allowed movement back and forth. By adjusting the distance between the display and the lens, the focal length was roughly measured by finding the point where the display appeared sharp to the eye. 4.4. Designing a Case - Iteration 4 19 (a) fresnel lenses with display (b) full test board Figure 4.4: (a) The display holders are designed to hold the Fresnel lenses in place without restriction. (b) To determine the focal length of the lenses, the mounting point for the displays can be adjusted, moving back and forth. With the approximate focal distance, the next iteration can implement a more reliable method to adjust it. The sliding display system was unfeasible, as the clamps that secure the displays to the floorboard were too fragile. To stabilize the lenses, a top piece would be required to support the upper holding bracket. This improvement would also improve the overall stability of the design, allowing for a mounting plate for the driver boards. A dedicated holder would be useful for future development. 3D printing is a viable solution to produce parts for a headset. This leads to fast development and more flexible prototyping with a sturdy build quality. Learning 3 4.4 Designing a Case - Iteration 4 With the introduction of iteration 4, the lenses were now fully encased, which locked them in place while also allowing swapping of lenses. Compared to the previous iterations, the displays are locked in place, and the lenses are now free to move and can be individually spaced. In addition, to improve stability and support the lens holder, a top cover was designed. A big issue with all designs was that the driver board was not held in place. With the introduction of a compartment at the bottom, the driver board can be inserted into it. In Figure 4.5, an CAD design of the headset is illustrated. Although this approach worked well for earlier prototypes, it was insufficient for this iteration. As the first version 26 Chapter 4. Iterative Hardware Design 4.6.4 IPD Adjustment For IPD adjustment, a separate mechanism was designed on top of the case, which connects to the lens module. As mentioned in Section 4.6.1.2, the outer shell of the lens module has a small knob on the top side that houses a threaded insert. This knob is the connection point for the mechanism. To reach it from the outside, a capsule formed hole was cut into the top of the hull. This allows a connection piece to attach to the knob and be secured with a screw. In addition, the connection piece has a linear rack attached to it, which can be moved using a rack and pinion mechanism. The gear on top acts as the dial to set the IPD. To lock the current adjusted IPD, a screw on the top can be fastened, which presses down on the attachment plate that holds the adjustment screw in place. The system can be seen in Figure 4.10 4.7 Discussion of Hardware In this section, hardware creation is discussed. Starting from the international process of creating the headset. Explaining how certain iterations impacted the final iteration and how the individual requirements were met in the end. Furthermore, the headset itself is compared to other consumer-grade headsets with respect to certain features. 4.7.1 Iterations In the previous sections, six iterations of the device have been presented. The process began with a simple cardboard model that did not have functionality. This non-functional prototype was created to gain a better understanding of the most significant challenges and obstacles. An issue that was initially underestimated was the adjustability of the distance between the display and the lens. Before building the prototype, this issue seemed minor, but during the process of assembling the cardboard box, it became clear that handling all requirements (see Table 3.1) would be more complex than expected. This early insight allowed the design process to be adjusted early on. The second iteration introduced a basic working prototype, which was still primarily made of cardboard. With the help of 3D printing, specialized components can be added to hold electronic parts, such as displays. This version enabled the first successful test of the stereoscopic effect. However, the prototype was still fragile. It could not support any tracking components and was not stable enough to be picked up or handled reliably. The third iteration eliminated the use of cardboard and employed only 3D-printed parts that were glued together. One of the main improvements in this version was the introduction of an adjustable mechanism for the display-to-lens distance. This was used to experimentally determine the correct focal length for the Fresnel lenses, which were also introduced in this version. However, while the lens adjustment was effective, the overall build remained very unstable. The glued parts could not hold their position under stress, 4.7. Discussion of Hardware 27 and the prototype was not durable enough for regular handling. With the measurements and learnings from the last iteration, the fourth version was created. It featured a more solid adjustable system for the display-to-lens distance, which was now within the usable range of the lenses. A compartment was also added for all electronics, allowing the entire device to be picked up and handled as a single unit. One major issue that remained was the use of glue. In real-world usage, especially with rough handling, the glued joints could not withstand stress and started to fail. This made the structure unreliable and highlighted the need for a better solution. In iteration 5, the form factor of the headset was improved. By introducing a detachable system on the back of the headset, the driver board compartment could be moved there. Furthermore, most of the components were new and were attached with screws, which significantly improved the stability of the system. The only components glued were the display and the breakout board holder. In Table 4.1, the individual learnings from each iteration are listed. These learnings were directly considered in the final design of the headset. The design consists of a modular system composed of four main parts: two eye modules, one compartment for additional hardware, and a central case that holds everything together. Each eye module includes mounting points for electrical components. In addition to housing the electronics, the modules also integrate the lenses. A key feature of these parts is the dial wheel mechanism, which allows the user to adjust the distance between the display and the lens. With this, requirements 1 and 3 are fulfilled, and the depth adjustment aspect of requirement 4 is satisfied. The electronics compartment is designed with a simple snap-on system, allowing the Learning 1 Creating a cardboard prototype is a feasible step for identifying potential obstacles early in development, which can lead to fewer iteration cycles, even if the cardboard solution is not ultimately successful. Learning 2 The displays should not move and be well seated or screwed, as this breaks user immersion and causes discomfort when wearing. Learning 3 3D printing is a viable solution to produce parts for a headset. This leads to fast development and more flexible prototyping with a sturdy build quality. Learning 4 Even on prototypes, a sturdy build is a must otherwise, the user needs to handle it very carefully, and problems that can arise from improper handling are not noticed. Learning 5 A sturdy and fixed build is essential for a headset to reduce wear and tear on the components and to provide a better experience for the user. Additionally, the display must be shielded from incoming light, as any light leakage breaks the illusion. Table 4.1: The learnings over each iteration, which were incorporated into the last design. 28 Chapter 4. Iterative Hardware Design Meta Quest 3 Valve Index PS VR2 Our Solution Release Year 2023 2019 2023 2025 Type Standalone PC Console PC Resolution per Eye 2064 ×2208 1440 ×1600 2000 ×2040 1600x1600 Max. Refresh Rate 120 Hz 144 Hz 120 Hz 90 Hz FOV ≈110°108°≈110°≤90 Tracking Inside-out Outside-in Inside-out Inside-out IPD Adjustment 53–75 mm 58–70 mm 58-73 mm 64-83 mm Weight 515 g 809 g 560 g 568g Fully Customisable No No No Yes Table 4.2: A comparison of existing headsets and our solution. The one thing to keep in mind is that the panels and lenses are swappable, this means some metrics are dependent on the configuration. The best values are highlighted in bold, while values that can be changed through swapping of components are underlined. module to be attached to the back of the case without tools. It can be customized according to the components used and satisfies requirement 2. The central case forms the core of the system. It contains the IPD adjustment system, which is another aspect of requirement 4. In addition, it includes dovetail rails to mount the eye modules and snap-on slots for the rear electronics compartment, making it the part that connects and holds the entire headset together. 4.7.2 Comparison of the Final Design In Table 4.2, the final hardware is compared to some existing commercial headsets. For this comparison, the Valve Index from 2019, the Meta Quest 3 and the PS VR2 were selected. These headsets span a wider range of release dates and specifications, which allows a better comparison of the prototype within the hardware that is on the market. The table lists individual features where the best values are highlighted in bold. Features that can be adjusted or customized are underlined to indicate their flexibility. Because technology is growing rapidly, the release years were mentioned to have a better reference point for comparison. Our solution is PC-based, meaning that a Computer needs to be connected to render the image, which is then displayed on the headset. For other headsets, the methods are console-based, where the headset is connected to a gaming console like the PlayStation, and Standalone, where the headset itself can render the image and does not need a separate device for this. Compared to other headsets, the resolution per eye is not comparable to that of Meta Quest 3. Our solution is significantly lower and more comparable to the Valve Index. The 4.7. Discussion of Hardware 29 reason for this is the off-the-shelf screens, which are not capable of providing a higher resolution. For the refresh rate, the Valve Index is the leader with 144 Hz, which is higher than most headsets on the market. Comparing this refresh rate with the Meta Quest 3 and the PS VR2, which both have a maximum refresh of 120Hz. The reason for this may be that the Valve Index has a significantly lower resolution than Meta Quest 3 and PS VR2. Our solution has a refresh rate of 90 Hz, where the panels are the limitation. The FOV cannot be measured correctly but is estimated to be less than 90°, which is not really a competitive solution. This value is influenced by the combination of the lens and the display, for example, a lens that has a smaller focal length influences the distance between the lens and the display, and the closer the display, the higher the FOV. Also, when the display is not wide enough, the FOV shrinks because the user can see areas where the display is not prominent. For tracking, an Inside-Out solution was easier to implement and used less setup than Outside-In tracking. Although inside-out tracking is more mobile, Outside-In tracking is more robust and accurate. In addition, IPD adjustment is an important feature of a headset, as incorrect IPD can cause discomfort to the user. While the Meta Quest 3 has the largest range, our solution can reach a higher maximum distance, but on the other hand, the minimum is quite high compared to other headsets. This can be explained by the width of the lens modules, which are 3D printed. The lightest headset in this comparison is the Meta Quest 3, with a weight of 515g. This is comparable to our solution, which has about 52g more. When compared to older models like the Valve Index, the whole system is much lighter, which is, of course, beneficial to the user. Although our headset is not outstanding in a specific feature, one thing that stands out among all other features is the aspect of customization. By being able to switch out a display for one with much higher resolution and better frame rate, the headset could compare or even outperform an Oculus Rift 3. With the option to swap out the lens, the system can also increase its field of view. By being able to swap out different electronics, the headset can be equipped with an Inside-Out tracking system or an Outside-In system, depending on the circumstances. This also means that the headset can be adjusted to accommodate a wider range of IPD adjustment. 5 Software implementation Contents 5.1 ProofofConcept............................ 31 5.2 Integration of OpenXR driver . . . . . . . . . . . . . . . . . . . . 37 5.3 Discussion................................ 53 5.1 Proof of Concept Data Transfer Processing Driver Receiver USB RGBD Camera RGBD Odometry Publisher Unity Subscriber Main Thread Dispatcher Update Pose Render Stereo Camera System Device Output Figure 5.1: Software structure of the initial prototype. This describes the path of odometry data from its generation at the RGBD device to its processing and rendering at the stereo camera system. The different colors describe different Software component types. At the bottom of the image, the meaning of the individual colors are displayed. The goal of the prototype was to create a knowledge base. It was built to achieve a first working prototype as fast as possible, so a majority of the code and technologies were 31 32 Chapter 5. Software implementation (a) Orbbec Astra (b) Orbbec Astra Embedded S (c) Structure Core ST02D-C (d) Intel RealSense D435i Figure 5.2: Cameras used for evaluation and comparison. not well-designed up front. In Figure 5.1, the whole pipeline is displayed starting from the top right corner, where the RGBD camera is capturing RGB and depth images. In the driver script, the system received the individual images, which are used to calculate a relative position change compared to the previous image data. The change in position is transformed into a transformation matrix, serialized, and sent over a publish/subscribe connection to a Unity application, where it gets received by a subscriber. Because Unity only allows interactions with its application through the main thread, a main thread dispatcher is introduced to forward the transformation data to the stereo camera implementation. The position of the individual cameras is transformed by the transformation matrix. This can result in a change in the rotation and position of the stereo camera system. Unity renders the individual frames in full-screen mode on the individual displays, allowing the user to view the rendering through the headset. In this section, each component of this prototype is described in more detail. 5.1.1 RGBD Camera Driver To obtain RGBD data, a suitable camera must be selected. Initially, a variety of RGBD cameras were tested in terms of their specifications. This includes the Orbbec Astra Embedded S, Structure Core ST02D-C, Orbbec Astra and the Intel Realsense D435i (see Figure 5.2). Some specifications can be seen in Table 5.1. Starting with the Astra and Astra Embedded S, there is no substantial difference Camera Model Depth Camera RGB Camera FOV (H×V) Weight Orbbec Astra 640×480 @ 30 fps 640×480 @ 30 fps 58.4°×45.5°310 g Orbbec Astra Embedded S 1280×800 @ 30 fps; 640×400 @ 60 fps 1920×1080 @ 30 fps 67.9°×45.3°27 g Structure Core ST02D-C 1280×960 @ 54 fps 640×480 @ 100 fps 49°×46°52.5 g Intel RealSense D435i 1280×720 @ 30 fps (up to 90 fps) 1920×1080 @ 30 fps 87°×58°72 g Table 5.1: Specifications of investiagted cameras 5.1. Proof of Concept 33 between them in terms of resolution and frame rate for the RGB sensor. However, the depth camera of the Astra Embedded S has a significantly higher resolution than that of the Astra. A major drawback of the Astra is its weight of 310g and dimensions of 165mm x 48mm x 40mm (width x height x depth), making it too large and heavy for a headset. In comparison, the Astra Embedded S, weighing just 27g with dimensions of 69mm x 22.9mm x 15mm, clearly outperforms the Astra in terms of size and weight. Deciding whether the Structure Core ST02D-C is a better choice than the Intel RealSense D435i depends on comparing their specifications. The RealSense depth camera has a framerate of up to 90 fps, a resolution of up to 1280x720, and a field of view of 87◦,×,58◦. The ST02D-C has a framerate of up to 60 fps, a resolution of up to 1280x960, and a field of view of 49◦,×,46◦. The RGB camera of the Structure Core ST02D-C can record at a resolution of 640x480 with 100 fps, while the D435i records at 1920x1080 with 30 fps. One key factor that was not mentioned is the compatibility and ease of use. The Intel Realsense D435i is widely supported throughout the industry and used by many technical companies. For this use case, the D435i was chosen over the Structure Core ST02D-C because of the ease of integration and native support with multiple libraries. The last decision was to choose between the Astra embedded S and the Intel Realsense D435i. The choice fell on the Intel camera not only because of its better software support but also because of its superior specifications. Although the Astra camera weighs 27g and the Intel camera 72g, the improved depth accuracy, wider field of view, and more robust SDK of the Intel RealSense D435i outweigh the benefit of a lighter device. With the camera fixed, the driver can now be written. To create a fast-working prototype, Python 1def getImage(pipeline, align): 2frames =pipeline.wait_for_frames() 3aligned_frames =align.process(frames) 4depth_frame =aligned_frames.get_depth_frame() 5color_frame =aligned_frames.get_color_frame() 6if not depth_frame or not color_frame: 7return [], [] 8 9depth_image =np.asanyarray(depth_frame.get_data()) 10 color_image =np.asanyarray(color_frame.get_data()) 11 return depth_image, color_image Listing 1: The following code retrieves an image from the camera and returns it as a NumPy array. First, the images are obtained from the pipeline. To compensate for the spatial discrepancy between streams, an alignment process is applied in line 3. Afterward, each image is checked to ensure it is not None. If either image is missing, a set of empty lists is returned, signaling the caller to skip the frame. If everything functions correctly, the images are converted into NumPy arrays and returned. 34 Chapter 5. Software implementation was used with the PyRealsense2 library. By connecting the RGBD device with a USB cable, the program can read the live image data and also the intrinsic camera parameters. Before the driver could retrieve the device images, a software pipeline had to be instantiated from the library. First, the USB connection to the device was established. Then, parameters such as the frame rate, the number of streams (in our case, two: one for RGB and one for depth images) and the depth scale had to be set. Afterward, the pipeline can be opened and the images can be retrieved. In the code snippet 1, the retrieval of depth and RGB images is demonstrated. First, both images are obtained from the device in line 1. Afterward, they are aligned, as they originate from different viewports and are slightly shifted. The alignment process corrects for this discrepancy. If one of the streams is empty, the entire frame is skipped, as indicated by the return statement on line 7. Finally, both images are converted into NumPy arrays and returned for further processing. 5.1.2 Pose Tracking The Open3D library was used for pose tracking. This enables us to estimate the movement and rotation between frames and therefore, allows us to transmit this data further on to Unity. Open3D has two algorithms implemented for use in odometry. In this case, the default value is sufficient because first of all this is a prototype which does not need to be efficient and second, the developers of Open3D already benchmarked the algorithms and concluded that the algorithm of Park et al. [45] takes as long as Steinbrucker et al. [46] but has more accurate results [47]. In Listing2, the Open3D function call is shown. However, before we can call this function, an RGBDImage object must be instantiated from the current data pair. With this object, we can call the function with the previous and current RGBD images. This results in an Open3D results object, which can be transformed into a numpy matrix. It represents the transformation matrix from the source frame to the target frame. By scaling the translation part of the matrix by 10, the movement in the x,y, and z directions is amplified for better visualization purposes. This matrix gets serialized to JSON notation and can now be transferred to Unity. 1result =o3d.t.pipelines.odometry.rgbd_odometry_multi_scale 2(source_rgbd_image, target_rgbd_image,o3d_intrinsics) Listing 2: Function call for RGBD odometry. The parameters from left to right are the RGBDImage object holding the previous frame, the RGBDImage object holding the current frame, and the camera intrinsics. 5.1. Proof of Concept 35 5.1.3 Interaction between Unity and Driver To establish data transfer between Unity and the driver, a simple networking approach was used. Message Queuing Telemetry Transport (MQTT) was chosen to send the data to Unity. The goal was to create a replaceable pose estimation system that could function independently of a specific driver. Thanks to MQTT’s publish-subscribe model, this is possible. Unity does not need to know the data source. It simply subscribes to a predetermined topic, allowing any connected client to publish data to the broker. 5.1.3.1 Driver integration In Python, a library called paho.mqtt handles the connection. After the client has been initialized, it connects itself to localhost, where the broker is hosted. The broker used is Eclipse Mosquitto. A simple Docker Compose file was created to start the broker. A small config file has been written to allow the broker to start. After establishing the connection between the driver and the broker, the data of the matrix can now be sent on to a predetermined topic. In this, the topic was called ”odometry”. 5.1.3.2 MQTTnet in Unity In the C# environment, the NuGet package manager provides an MQTT library called MQTTnet. However, Unity does not natively support NuGet packages. Fortunately, a Unity plugin called NuGetForUnity [48] enables the installation of NuGet packages, including MQTTnet [49]. By instantiating an MQTT client and specifying connection parameters, such as IP address, port, and client name, the program can connect to the broker. When subscribing to a topic, a callback function must be provided that can handle the data. During initial testing, a simple transformation was applied to a cube, but the function call did not modify the cube. The reason for this issue was that the function was not executed from the Unity main thread. To resolve this problem, a separate Unity object was created with a Main Thread Dispatcher script attached. This script allowed tasks to be appended to a queue, and each time Unity’s Update function was called, a task was dequeued and executed within the main thread. This allowed the function to be scheduled to the Unity main thread. From there, the data could now interact with objects in the Unity scene. 5.1.4 Unity integration The first challenge was to configure two cameras to display on separate screens using Unity. The scripting API allows calls to the variable Camera.targetDisplay, which defines the target display to which the camera is rendering. To establish a common rotation point, which serves as the camera’s point of origin, both cameras are placed within the same parent. The script for setting the camera target 42 Chapter 5. Software implementation 5.2.3 Building a VI-SLAM library on Windows Now that the runtime supports loading the interface, a VI-SLAM library must be built on Windows to generate a DLL that the system can load and interact with. There are many different libraries and methods available that implement some form of visual-inertial tracking. After several failed attempts with different libraries, ORB-SLAM3 showed promising progress during the compilation process. After running CMake on the project for the first time, many errors occurred, most of which were compiler-related. MSVC does not recognize many build flags used by the Linux compiler. This led to the removal of several CMake definition lines and the deletion or replacement of flags that MSVC could not interpret. There were also some dependency issues involving multiple libraries. Some of them were built directly in the project, for example, DBoW2, which implements a hierarchical bag-of-words approach to find similar images. Other applications were expected to be installed on the system, such as the RealSense library, which enables access to the RealSense cameras. Some of these issues were resolved by changing the file extension from .so to .dll. Other problems were resolved by installing missing dependencies or successfully compiling the required projects. After addressing all errors, the system generated the CMake files needed to build the entire project. The next step was to build the system. As expected, this resulted in additional errors. Due to the difference in operating systems, the usleep() function was not defined. Additionally, the C interface of the library had to include the declspec(dllexport) attribute for its exported functions. There were further issues that needed to be fixed, but in the end the system successfully built the DLL file. Unfortunately, when testing it in the wrapper library, it did not work due to a bug in the code, likely caused by compiler differences. The exact cause of the failure was not further investigated and the ORB-SLAM3 project was ultimately not used. This process of fixing the compilation and then encountering a major roadblock happened multiple times. Ultimately, the Basalt library was utilized, which is also recommended by the creators of monado. Fortunately, Basalt implements the VIT interface, making the previously selfwritten wrapper library obsolete. As in the previously described build, the first step was to fix the project’s CMake files. The C++ standard was set to C++20 to ensure that all required language features and standard library classes are available. Additionally, the compiler flag /bigobj was added to increase the maximum number of sections in an object file from 216 (65.536) to a higher limit, thereby avoiding compilation errors. Next, several lines in the build configuration had to be adjusted because they included parameters not supported by MSVC. For example, the -O3 flag used in Linux enables more aggressive optimizations than -O2, but it is not available in the Windows compiler and causes an error. To resolve this, -O3 was replaced with /O2, ensuring that the compiler would still 5.2. Integration of OpenXR driver 43 optimize the code. To manage dependencies, the VCPKG package manager was integrated into the project. This allowed all required libraries to be automatically installed and correctly located by CMake. Additional dependencies were included in a third-party directory containing projects that are also built as part of the root Basalt project. After resolving these issues, CMake successfully generated the required build files. However, during the build process, additional issues arose. One minor problem was the redundant use of the template keyword on non-template objects. For example: Q2Jp.template block(start_idx, 0, poses_size, poses_size) In this case, the variable does not require the template keyword. Simply removing it in multiple instances resolved the compilation errors. Another issue involved undefined macros that were expected to be provided by the standard library. One such case was the missing definition for the maximum float value, typically referred to as MAXFLOAT. A macro named FLT MAX already has this value, so the line #define MAXFLOAT FLT_MAX was sufficient to fix the problem. Additionally, the type SIZE T was undefined. This was resolved by adding a typedef, defining SIZE T as an alias for size t. It also became evident that MSVC is stricter with implicit casts, so several of these had to be replaced with static cast to prevent compilation errors. Another issue related to standard library differences was a name collision. A class implemented two functions named min and max, which conflicted with macros of the same name defined in the standard library. This resulted in confusing error messages that stated that the min and max functions declared in the header file were undefined. The solution was to #undef the min and max macros in that file, which resolved the conflict. In one particular file, the Anonymous Pro font was used by defining a constant variable that was supposed to hold the binary data from a separate C header file. However, this file was missing from the repository. The font had to be downloaded manually from Google Fonts. Since no code was available to load a .ttf file directly, the entire binary content had to be transferred to a C header file. To achieve this, the xxd command was used to convert the binary .ttf file into the target file format. This process creates a large array that contains the binary data of the font as a const unsigned char variable. The command 1xxd -i AnonymousPro.ttf > AnonymousPro.inc 1extern "C" const unsigned char AnonymousPro_ttf[] ={ 20x00,0x01,0x00,0x00, ...} 3unsigned int AnonymousPro_ttf_len = 112280; Listing 8: xxd is used to create a hexdump of any file. In this case, it converts the font into a C header file containing the binary data. The -i flag enables C-style array output. On the bottom, we can see the definition and size of the array. 44 Chapter 5. Software implementation used for the conversion is shown in Listing 8. The resulting header file was then included in the appropriate namespace, resolving the font-related issue. The final failure point involved the export of the VIT interface. As mentioned previously, exporting functions in a DLL requires explicit marking. Once all relevant functions were marked correctly for export, the entire compilation process was completed successfully and Monado was able to load the DLL and instantiate a tracking object. 5.2.4 Adaptation of the RealSense Driver In Monado, a driver for detecting and registering a new RealSense camera was already implemented. The implementation is split into two different modes. One in which the host system calculates the pose and one in which the camera itself returns tracked positions. In this implementation, the first system was used. Fortunately, the implementation was fairly general, making the integration of the camera straightforward. However, some default parameters needed to be adjusted. These values are shown in Listing 9. A notable parameter appears in line 2, where the camera’s video format is set to RS2 FORMAT Y8. This indicates that the stream delivers an 8-bit grayscale image, which can be retrieved from the streams defined by the indices in lines 16 and 17. Additionally, in line 15, the stream type is set to the infrared camera, which aligns with the default video format specified for the RealSense device. 1#define DEFAULT_STEREO true 2#define DEFAULT_XRT_VIDEO_FORMAT XRT_FORMAT_L8 3#define DEFAULT_VIDEO_FORMAT RS2_FORMAT_Y8 4#define DEFAULT_VIDEO_WIDTH 848 5#define DEFAULT_VIDEO_HEIGHT 480 6#define DEFAULT_VIDEO_FPS 30 7#define DEFAULT_VIDEO_CHANGE_EXPOSURE true 8#define DEFAULT_VIDEO_AUTOEXPOSURE false 9#define DEFAULT_VIDEO_EXPOSURE 33000 10 #define DEFAULT_VIDEO_GAIN 16 11 // IMU Settings 12 #define DEFAULT_GYRO_FPS 200 13 #define DEFAULT_ACCEL_FPS 200 14 // Stereo Stream settings 15 #define DEFAULT_STREAM_TYPE RS2_STREAM_INFRARED 16 #define DEFAULT_STREAM1_INDEX 1 17 #define DEFAULT_STREAM2_INDEX 2 Listing 9: In the camera driver, the default values were targeted toworads the RealSense D455. The default parameters did not coincide with the D435i therefore the parameters have been replaced with the needed default values. 5.2. Integration of OpenXR driver 45 The default frequency for both the gyroscope and the accelerometer is set to 200 Hz. This value is specifically chosen on the basis of the camera’s capabilities. Similarly, the resolution was set to 848×480 at 30 frames per second. Like the IMU frequency, these values are selected according to the camera’s supported specifications. In order for Monado to correctly interpret and handle the incoming data, a video format must be specified using a Monado-specific format type. In line 2, the format is set to XRT FORMAT L8, which defines an 8-bit luminance channel. This format is interpreted as an RGB image where each pixel has the same value across all color channels. By aligning the camera’s video format with Monado’s expected format, the system can correctly process the incoming data. All other parameters,such as gain and exposure,were adjusted using the RealSense Viewer. This tool allows live parameter tuning while observing the camera stream. Through trial and error, appropriate values for gain and exposure were determined. Another change was the acceptance of the Windows Slam Tracking system. This change involved the addition of the XRT SLAM WINDOWS to the check prior to instantiation of the slam system. As a final step, the function responsible for creating a driver instance was exposed in the driver’s library. This function is typically used by the auto-prober, which detects new devices and creates the corresponding instances. 5.2.5 Implementing the Target Builder Now that the device driver has been written, the RealSense driver has been adapted, and the SLAM library has been integrated, the final implementation step in Monado can be completed. This involves adding a new file named target builder custom vr.c to the src/xrt/targets/common directory. The entire system for creating a new device in Monado is built around a builder pattern. This approach enables modularity and extensibility, allowing for the support of a wide range of devices. To inform the program about how to create the VR device, the function t builder custom vr create must be defined. This function is responsible for returning a preconfigured builder object that is then used to instantiate the device. Additionally, the function must be registered in the target list variable and wrapped in a #ifdef block to prevent compilation errors when the corresponding CMake flag is not set and the driver is not included in the build. The builder function is seen in Listing10. This code block defines the builder function. In line 3, the builder struct is instantiated. Line 5 sets the function responsible for estimating the system components, allowing Monado to know the features of the device without opening it. In line 6, the function for opening the device is assigned, while line 7 sets the function for destroying the device instance. Lines 8–11 define several metadata fields that can be printed when using the CLI. Line 12 provides the option to exclude the device from automatic discovery through an environment variable. Finally, line 15 implements the actual 46 Chapter 5. Software implementation 1struct xrt_builder*t_builder_custom_vr_create(void) 2{ 3struct u_builder *ub =U_TYPED_CALLOC(struct u_builder); 4 5ub->base.estimate_system =custom_vr_estimate_system; 6ub->base.open_system =u_builder_open_system_static_roles; 7ub->base.destroy =custom_vr_destroy; 8ub->base.identifier ="custom_vr"; 9ub->base.name ="Custom VR devices builder"; 10 ub->base.driver_identifiers =driver_list; 11 ub->base.driver_identifier_count =ARRAY_SIZE(driver_list); 12 ub->base.exclude_from_automatic_discovery = 13 !debug_get_bool_option_enable_custom_vr(); 14 15 ub->open_system_static_roles =custom_vr_open_system_impl; 16 17 return &ub->base; 18 } Listing 10: These lines of code define the fields for the XR device builder. First, the builder object is instantiated. Then, the functions for offline feature estimation, device opening, and builder destruction are assigned. Following that, several debug fields are defined to provide useful information during runtime. The last two fields handle exclusion from automatic device discovery and specify the function responsible for instantiating the actual XR device. instantiation of the device. This function is implicitly called by open system. The function custom vr open system impl is responsible for creating the device instance. In Listing 11, the most important lines are shown. In line 1, the custom VR headset instance is generated. The called function is implemented in sample hmd.c. In line 2, the SLAM device, specifically, the RealSense driver, is instantiated with a prober attached 1struct xrt_device cv_device =custom_vr_create(); 2struct xrt_device slam_device =create_tracked_rs_device(xp); 3struct xrt_device head_wrap =multi_create_tracking_override( 4XRT_TRACKING_OVERRIDE_DIRECT, 5cv_device, slam_device, 6XRT_INPUT_GENERIC_HEAD_POSE, 7&head_offset); 8ubrh->head =head_wrap; Listing 11: These lines of code combine two XR devices in order to apply tracking to the custom VR headset. First, the custom VR device and the tracking device are instantiated. Second, the tracking functionality of the custom VR device gets overwritten with the tracking device. Then, the resulting object is assigned to the desired role. 5.2. Integration of OpenXR driver 47 that can detect whether the device is connected. The RealSense implementation library exposes this function. To combine the tracking capabilities of the RealSense driver with the functionality of the custom VR headset, the function in line 3 is called. It is responsible for overriding the necessary function pointers of the custom VR device to enable external tracking integration and ensure that both systems work seamlessly. In line 4, the parameter for directly overwriting the headset’s tracking is set. There is also an alternative mode in which the system enters an ”attached” state, allowing the internal tracking of the VR headset to coexist with the RealSense tracking. However, this is not the case here, so the direct overwrite method is used. Line 5 specifies both the base device to be overridden and the SLAM device that provides the tracking. Line 6 defines the pose type as a head, and in line 7, the positional offset to the user’s head is passed to the function. This call creates a new device that combines both visual and tracking components, which is then assigned as the head device on line 8. The final functionality that was implemented in the builder is the definition of the offline feature estimation. This is shown in Listing 12. In lines 1 and 2, the expectation that a head-mounted device will be present is explicitly defined, and the maybe flag is also set to true. Lines 4 to 7 specify that hand tracking will not be available. Lines 9 to 12 determine whether 6 Degrees of Freedom (6DoF) tracking will be supported. This depends on whether the RealSense D435i is connected to the computer. The function call on line 9 checks for the presence of the device by verifying whether any connected device matches the predefined Product ID (PID) and Vendor ID (VID). These values are 0x8086 for the VID and 0x0B3A for the PID. They are defined as macros in the implementation of the RealSense driver. 1estimate->maybe.head =true; 2estimate->certain.head =true; 3 4estimate->certain.left =false; 5estimate->certain.right =false; 6estimate->maybe.left =false; 7estimate->maybe.right =false; 8 9estimate->certain.dof6 =u_builder_find_prober_device( 10 xpdevs, xpdev_count, 11 REALSENSE_D435I_VID, REALSENSE_D435I_PID, 12 XRT_BUS_TYPE_USB); Listing 12: This code defines the expected features for the device. The presence of a head is certain, while hand tracking is not supported. The most notable feature is 6DoF tracking, which is only enabled if a RealSense camera is detected using the predefined PID and VID. 48 Chapter 5. Software implementation 5.2.6 Unity Project Setup Figure 5.6: In this image, we can see the Unity project. It consists of multiple objects that are necessary for the OpenXR integration and platform, with cubes on top to gain a sense of space when using it. With the system up and running and the SLAM system ready to track the position, a test application can be built. Fortunately, utilizing the OpenXR standard makes this a well-documented task. For this test, a Unity project was set up because it provides full control over the scene, which can be beneficial since some applications restrict access to certain areas, leading to a more challenging tracking of bugs. To start a new application, a new Unity project was generated with the standard 3D project template. After project creation is complete, the XR management plugin can be installed, a package that provides simple management for XR plug-ins. It handles loading, initialization, and settings, and it also offers build support for XR-related components. After installation, new options are available. For the test, a new desktop application was used, therefore, the checkbox for OpenXR was ticked in the desktop section. This installed the plugin for OpenXR compatibility and changed the input system. Afterward, a restart of the project was necessary. After restart, the project identifies certain issues raised by the OpenXR system. Fortunately, Unity has a project validation feature that can automatically fix the issues. Only one issue was not fixable by the system, that was the selection of the interaction profile. This sets the bindings for a specific controller type, for example, an Oculus or an HTC Vive controller. In this project, this would not be necessary because we do not have a controller. However, for completeness, the ”Khronos Simple controller” was added because this name best aligns with the OpenXR 5.2. Integration of OpenXR driver 49 Figure 5.7: This window shows the rendered scene for each eye. theme. To access the necessary components in the editor, the package ”XR Interaction Toolkit” must be installed in Unity’s package manager. This installation includes components that interact with the OpenXR system. The project is now configured to support OpenXR, but it still requires the components to be instantiated and configured in Unity. To create a camera for rendering, a new XR Origin component was created. This component represents the player and holds the interaction with the OpenXR system for the head. With this setup, the project can be started. Through the OpenXR load, the system can locate the OpenXR runtime and register the application with it. Monado is currently configured to open a window that corresponds to the resolution of the displays. The resulting unity project can be seen in Figure 5.6resulting image can be seen in Figure 5.7. 5.2.7 Display writeout This window, as seen in Figure 5.7, is a single window, which is not what we wanted because we have two displays. In Linux, this problem would be easily solved. The tool xrandr can be used to software-fuse two displays, making them appear as one large display. Monado can then use this display to display the rendering in fullscreen. This would be the full-screen approach. There is also a technology called Nvidia Direct Mode, which can directly display the 50 Chapter 5. Software implementation rendered content onto displays, saving performance by reducing copying and transfer operations. This would also improve the device’s latency. But with the limitation of having two separate displays, this kind of rendering is not possible. This means that Monado needs to supply two different windows and support direct mode for two displays simultaneously. Unfortunately, Monado is built in such a way that adding another window with its own swapchain and window management is not a trivial task. After some trial and error, no significant progress could be made, leading to the conclusion that knowledge and experience in programming real-time rendering pipelines were insufficient. This leads to the option of an exterior solution. In general, there were two options: one was to copy the contents of the window and display them in two separate windows. The other was to adjust the displays in the virtual display space. This means that we place the two displays side by side and move the window over both displays. In the following subsections, the implementation details are shown. 5.2.7.1 Copying the Output The idea behind this approach was to use a virtual display to render both views in full screen. This full view can then be copied, and Windows recognizes the display screens built into the device. To achieve this, virtual display driver (VDD) [51] was downloaded and installed. As the name suggests, this driver allows the creation of a new virtual screen that can be freely configured. In this case, the resolution of the virtual display was set to match the height and double the width of one of the built-in panels. Once Windows recognized the display, an environment variable was set before launching Monado. This variable ensured that the rendered full-screen view would be shown on the virtual display. With that in place, it became possible to capture the image data directly from the screen. In Listing 13, the memory setup for the screen grabber is shown. To keep the copy process efficient and reduce unnecessary data movements, the system was configured to perform only one copy operation per screen refresh. This helps reduce latency, which is critical in a real-time system like this. In the first line, a handle is retrieved for the entire screen. To map the captured screen content directly into an OpenCV Matrix object, a Device Independent Bitmap (DIB) section is utilized. This section returns a pointer to the allocated memory. But before the DIB section can be created, a BITMAPINFO object must be set up. This is done between lines 3 and 12. The most notable attributes in the definition are the height and the bit count. To capture the screen starting from the top, the height is set as a negative value. This inverts the image vertically, so the top of the screen matches the top of the image buffer. The bit count is set to 24, which means that only the red, green, and blue channels are used. In this case, no alpha channel is needed. 5.2. Integration of OpenXR driver 51 1HDC hdcScreen =GetDC(NULL); 2 3BITMAPINFOHEADER bi ={}; 4bi.biSize =sizeof(bi); 5bi.biWidth =width; 6bi.biHeight = -height; 7bi.biPlanes = 1; 8bi.biBitCount = 24; 9bi.biCompression =BI_RGB; 10 11 BITMAPINFO binfo ={}; 12 binfo.bmiHeader =bi; 13 14 void *pBits =nullptr; 15 HBITMAP hDib =CreateDIBSection(hdcScreen, &binfo, 16 DIB_RGB_COLORS, &pBits, NULL,0); 17 18 HDC hMem =CreateCompatibleDC(hdcScreen); 19 SelectObject(hMem, hDib); 20 21 int stride =((width*3 + 3)& ~3); 22 cv::Mat screen_mat(height,width,CV_8UC3, pBits, stride ); 23 Listing 13: In this code block, the memory for the copy process is prepared. In the first line, the handle for the screen is retrieved. To perform the copy operation, a Device Independent Bitmap (DIB) is used, which requires a bitmap definition. This definition is created from line 3 to line 12. After that, the DIB section is created and connected to the device memory (lines 15–19). In the final two lines, the stride is set, and the OpenCV Matri is instantiated with a pointer to the allocated memory. After the info object is prepared, the DIB section is created in lines 15 and 16. It uses the hdcScreen and the binfo structure to define the memory layout. The value DIB RGB COLORS specifies that the memory should store RGB values. The NULL argument indicates that the memory should be stored in system RAM, making it fast and easy to access for subsequent processing. The last zero defines the offset in memory, which is none. To access the memory, the function call writes the memory address into the pBits variable. Now that there is a CPU-side memory that can be accessed, a new memory handle gets initialized for the screen device. Afterwards, the DIB is selected into the memory context, which ensures that any copy operation is directed to the DIB. Because the rows of the copied memory are aligned in multiples of 4, OpenCV’s matrix needs to be aware of this and act accordingly. This behavior can be set in the constructor of the matrix by specifying the last value. The stride variable gets set to the nearest 6 Building a VR Headset Contents 6.1 Assembling the Headset . . . . . . . . . . . . . . . . . . . . . . . 59 6.2 Compiling Monado for Windows . . . . . . . . . . . . . . . . . . 63 6.3 Starting the System . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.4 Configuring Environment . . . . . . . . . . . . . . . . . . . . . . . 66 This chapter provides instructions for creating the custom VR headset described in the previous chapters. It begins by listing the required parts for each assembly step and provides detailed instructions for building the hardware. A description of the compilation process for Monado follows this. Since Monado does not run as expected by simply starting the program, the necessary environment variables are listed, along with a detailed explanation of the user interface, allowing for custom modifications if needed. Finally, if game support via SteamVR is desired, this chapter also includes an explanation of the required modifications. 6.1 Assembling the Headset To create a new headset, a 3D printer, a Metal Saw, a lighter, or other source of heat, and a soldering iron are required. In Table 6.1, the parts that need to be purchased are listed. For 3D printing, a total of 39 parts need to be printed, 10 for each eye module, 8 for the case, and 1 for the electronics compartment. The assembly process is divided into two parts. The first describes the construction of the left eye module, which can also be used for the right eye module. For the second part, the full assembly of the case with the eye modules and the electronic compartment is described. 59 60 Chapter 6. Building a VR Headset PartN Count Product ID Threaded Rod M3 55mm 4 B0D5DGMXNN (Amazon) Press Insert M3 4 B0CRYWCYMG (Amazon) Allen Screw M2 4mm 40 B0CKN2VK12 (Amazon) Press Insert M2 41 B0CRYWCYMG (Amazon) Allen Screw M2 6mm 1 B0CKN2VK12 (Amazon) Display with Breackoutboard 2 1005008477815602 (AliExpress) MIPI-to-HDMI Board 2 1005008477815602 (AliExpress) Table 6.1: Components which need to be bought with their corresponding Product ID and vendor 6.1.1 Eye Module In Table 6.2, the required parts for the headset are listed, and in Figure 6.1, the 3D printed parts are listed and enumerated. The first step is to prepare the threaded rods. Using a metal saw, cut a 3mm deep line in the center of the front side of the rod. This cut divides one end of the rod into two halves. Remove one half so that a 3 mm high semicircle remains, connected to the rest of the threaded rod. This semicircle is used to connect to part 5 and transfer the rotational movement from the gear to the threaded rod. By heating the cut end, the rod can be pressed into the semicircular hole in the center of part 5. The heated metal will slightly melt the plastic, allowing it to deform and form tightly around the metal for a secure fit. This process is done twice, once for each part 5. The press inserts can then be mounted on the 3D-printed parts. It is important to press them straight, otherwise the screws may sit crooked or, in the worst case, not fit at all. Parts 1 and 2 each require four M2 inserts, marked by small holes on the bottom and the side. Part 3 requires two M2 and two M3 inserts: M3 inserts for the threaded rods and M2 inserts for securing the breakout board. Part 8 requires six M2 inserts, three in the back for the MIPI-to-HDMI board, two to hold the side cover, and one in the top for the connection to the IPD mechanism. With the preparations completed, the assembly can begin. First, the display is slid into the bracket in part 3 and the breakout board is mounted on the back (see Figure Part Count 3D Printed Parts 10 Threaded Rod M3 55mm 2 Press Insert M3 2 Allen Screw M2 4mm 15 Press Insert M2 16 Display with Breackoutboard 1 MIPI-to-HDMI Board 1 Table 6.2: Parts for one Eye module 6.1. Assembling the Headset 61 1 2 3 4 5 5 7 7 6 810 9 (a) Inner Module (b) Outer Module Figure 6.1: 3D printed Parts for the eye module with an enumeration 4.8 a and b). The display and breakout board are then connected through the opening in part 3. Next, part 2 is prepared by placing the lens inside, and part 1 is placed on top of the lens. Part 2 must be positioned below part 1 because it has a small cutout to route the connection cable from the breakout board to the outside (see Figure 4.8 c ). The prepared part 3 is then slid into parts 1 and 2 using the dovetail slides. The connection cable passes through the slit, and part 4 is mounted on the back of parts 1 and 2 to secure the assembly, using four screws. Now, screw both part 5 gears with the threaded rod into the two holes of part 4, which should align with the M3 insert in part 3. Now, screw the gears in until the gears sit flush on part 4. Also, make sure that part 3 is not tilted and is parallel to the lens. Now place part 6 on the middle bump of part 4, and one part 7 on the bump above. It should now look like Figure 4.8 d). The parts of the surrounding shell are shown in Figure 6.1 b. This shell encloses the assembled module, blocks out external light, mounts the HDMI-to-MIPI board, and connects the module to the main case. This is the most challenging part of the assembly process. The goal is to achieve the construction shown in Figure 4.9 a. First, part 10 is placed in the smaller compartment in the back of part 8 and secured through the hole with the second part 7 gear. The previously assembled module is then slid into the prepared part 8 with all gears already in place. During sliding, the cable must be routed through the opening on top of part 8. Once the module is almost fully inserted, there is a good chance that the part 7 gear holding part 10 will not align with the gears on part 4. By gently moving part 10 while pressing the assembly together, part 7 will engage correctly with the inner gear system. Once aligned, the assembly is secured in place by inserting two screws into the top and bottom front of part 8, which are positioned directly above and below the lens. Then Part 9 is placed and screwed onto the side of the module, sealing the shell. At this stage, 62 Chapter 6. Building a VR Headset Part Count 3D Printed Parts 9 Allen Screw M2 4mm 10 Press Insert M2 9 Allen Screw M2 6mm 1 Table 6.3: Parts for one Eye module the assembly should resemble Figure 4.9 b. For the final step, the HDMI-to-MIPI board is mounted to the back of the assembly, and the cable from the breakout board is connected. The completed module should now look as shown in Figure 4.9 c. The eye module is now fully assembled and ready for use. To build the module for the opposite side, the same steps are followed, but the process is mirrored. 6.1.2 Case and Electronics Compartment In Table 6.3, the required parts for the case are listed, and the required 3D printed parts are enumerated in Figure 6.2. As with previous assemblies, the first step is to prepare the parts for mounting. This means inserting all necessary threaded inserts before doing any other work. In this case, only part 11 requires threaded inserts. Four inserts are needed on each side to mount the side panels, and one insert is placed on top, centered on the bump. With this, the preparation for the case assembly is complete. First, the two eye modules are inserted into the left and right sides of part 11. To secure them, the left and right panels (parts 12 and 13) are mounted on their respective sides. It is important to check that both modules can move freely and do not bind to the dovetail slides during adjustment. 11 16 14 17 18 13 12 15 19 (a) Case Parts (b) Electronics Compartment Figure 6.2: 3D printed Parts for the case and the Electronics Compartment 6.2. Compiling Monado for Windows 63 For the IPD adjustment mechanism, part 17 is placed on top of the device. The higher side is inserted through the left hole and onto the top of the left eye module, where a hexagonal knob serves as the mounting point. The same process is repeated on the right side with part 18. Each connection point is secured with an M2 ×4 mm screw. After moving both modules so that they meet in the center, part 14 is placed on the center bump at the top of the case, enabling synchronized movement of both modules by turning the gear. Next, part 15 is inserted and fixed using an M2 ×6 mm screw, which is threaded into the prepared insert—this is the only step that uses a different screw size. Then, part 16 is screwed into part 14 so that, when tightened, it presses against part 15 and creates a friction joint. Because part 15 has a hexagonal connection point, it cannot rotate, which ensures that the entire IPD mechanism is locked in place. The mechanism should now look like Figure 4.10 a. Part 19 can be clipped on the backside of the device. It features two holes in which a camera can be mounted. The headset is now fully assembled and can be plugged into a PC with HDMI and USB cables. 6.2 Compiling Monado for Windows The first challenge to overcome is compiling Monado for Windows. For this, the Visual Studio 2022 compiler (MSVC) and the CMake build system were installed and used. For dependency management, VCPKG should be used. It is a cross-platform package manager for C++ dependencies. Fortunately, VCPKG is fully integrated into the project, so CMake handles dependency management. The required dependencies for Windows are illustrated in Table 6.4. With this in place, CMake successfully generates the project files, allowing the build process to begin. However, this leads to compilation errors, as the entire project was originally based on a Linux compiler. Unfortunately, the standard libraries differ slightly, which caused the initial compilation to fail due to M PI being undefined. This issue can be Dependecie name Description Eigen 3.x linear algebra library Pthreads Windows POSIX threading header for Windows SDL access to mouse, keyboard, and other peripherals Vulkan Graphic API HIDAPI Bluetooth and USB interface LibUSB For USB device access cJSON JSON parser for C WIL For easier Windows development Table 6.4: This Table lists the required dependencies for Monado along with a brief description of their general purpose. These dependencies must be installed to allow CMake to generate the necessary build files successfully. 64 Chapter 6. Building a VR Headset Figure 6.3: This image shows the monado UI consisting of multiple windows. quickly resolved by adding the line: #define M PI (3.14159265358979323846). After this addition, the compilation process completes successfully, and the system can be started for the first time. 6.3 Starting the System By building and executing the ”monado service” target, the service can be started and used. However, at this stage, neither a device instance is created nor is the camera detected correctly. The reason for this is the absence of certain required environment variables (see Table 6.5). The first variable is responsible for creating an instance of the custom VR device on startup. This is achieved by disabling automatic probing and implicitly registering the device in the system. The second variable is required for the RealSense driver. To correctly identify the connected cameras, the prober must be assigned a valid source index. Without this, the system crashes during startup due to missing stream configuration parameters. For the SLAM library, two additional environment variables are necessary. One defines the path to the SLAM library, in this case the Basalt library. The other specifies the configuration file for the D435i. Basalt requires this configuration file to operate correctly, as it contains essential parameters such as camera intrinsics, resolution, IMU biases, standard deviations, and other calibration data. These values can be extracted from the camera itself using the RealSense camera calibration tool. For the IMU-specific statistical values, a pre-generated configuration file tailored for the D435i was used. The last envi- 6.3. Starting the System 65 Name Value CUSTOM VR ENABLE 1 RS SOURCE INDEX 0 SLAM CONFIG ./d435i.toml VIT SYSTEM LIBRARY PATH ./basalt.dll XRT DEBUG GUI 1 Table 6.5: Environment variables for starting monado properly. The first one is for instantiating the headset. Second for the streaming index of the camera, third and fourth for the slam configuration and the last one to enable the debug gui. ronment variable enables the use of the debug GUI, shown in Figure 6.3. This interface was created using ImuGUI and provides controls and statistics for various features within Monado. Several windows within the UI are handy and have been frequently utilized during development. One notable window is located in the bottom left corner. It serves as the control interface for the video source inside the RealSense driver. The top of the screen displays the real-time values from the gyroscope and accelerometer. A toggle button on the side enables a graph view that visualizes these sensor values over time. Below that, infrared images from the left and right cameras are shown. In the top middle section, the RealSense Device window displays the status of the instantiated RealSense device. This window represents the RealSense driver implementation, which also handles the initialization of the SLAM instance. It provides real-time information about the currently tracked position and orientation of the device. Additionally, it allows configuration of translation and rotation offsets. A notable detail is the label at the top of the window, which states ”Tracked by: Host SLAM”. This confirms that the tracking is handled by the external SLAM library, rather than assuming that the RealSense device is performing tracking on its own. This detail is important because if the system says that it is device-tracked, the whole tracking would not work. After all, the D435i does not support internal tracking. On the right side, there is a large window with the title ”SLAM Tracker #1”. As the name suggests, this is the control window for the slam tracker. To start the whole slam tracking, the first checkbox needs to be ticked. This will start to route the data to the SLAM library. Right below, there is a reset button for resetting the tracker state. This is especially useful for debugging and testing the tracker when numerical instabilities occur. Further down, the section for filtering of the signal is seen. For all filters, the predefined values were used. The first filter is the moving average filter. This filter takes the average over the predefined window and uses this value. The exponential smoothing filter smooths the data in a way that recent data is given more weight than older data. The last one, the one euro filter, is a low-pass filter with an adaptive threshold. In the prediction section of the window, the current values of the accelerometer and 66 Chapter 6. Building a VR Headset gyroscope are displayed. These values are received from the RealSense driver. Also, the constant value for the gravitational correction is adjustable from this window. But one of the most interesting settings is the ”Prediction Type”. In this dropdown menu, there are multiple options: ”None”, ”Interpolate SLAM poses”, ”Also gyro”, ”Also accel”, and ”Latest IMU”. The ”None” option disables the entire prediction system. ”Interpolate SLAM poses” uses only the tracked poses from the SLAM system, and if intermediate poses are needed, the system interpolates between them. With ”Also gyro” and ”Also accel”, the data from the gyroscope or accelerometer, respectively, are added to the evaluation pipeline. The ”Latest IMU” option uses all available IMU data to retrieve the most accurate and up-to-date pose. In the last section, we can see another preview of the received image frames which are processed by the SLAM system. The middle bottom window, called ”Compositor #1,” can display useful information about the system’s performance. A critical value is the current frame rate, expressed in frames per second (FPS). This indicates the number of images that Monado can process in one second. Further down, the chart illustrates the time it takes for a new frame to be fully rendered and displayed. The last two sections, ”View[0]” and ”View[1]” show the current rendered image if an OpenXR application is running. 6.4 Configuring Environment To inform OpenXR runtime that our platform exists and that we intend to use it as our standard value, the registry must be edited. In Computer\HKEY LOCAL MACHINE \SOFTWARE\Khronos\OpenXR\1the path to the file openxr monado-dev.json must be set. This file is generated during the application build and resides in the binaries directory. It has all the necessary values for the OpenXR loader to load an OpenXR application with the Monado runtime. With this setup, it is possible to start VR programs from Unity, which then are displayed by Monado. For Steam gaming, there is a bit more setup involved. First of all, the Steam plugin needs to be compiled. This is done by compiling the ”driver monado” target. This creates a new file in the steamvr-monado/bin including the monado driver.dll which is the plugin for Steam. With this plugin, the driver now needs to be registered on Steam. This is done with a separate batch file that is shipped with the application. ./steam/steamapps/common/SteamVR/ ⌋ bin/vrpathreg.sh adddriver C:/path/to/dll/monado\_driver.dll let Steam know where the custom VR driver is. The success of the command can be verified by running vrpathreg.sh without any parameters. This enables the use of all hardware supported by Monado on Steam, but does not allow for playing games. To enable game support, the OpenComposite application must be installed. It is responsible for forwarding the SteamVR calls to OpenXR. wBy starting the program and 6.4. Configuring Environment 67 clicking on the ”Switch to OpenVR” button, the redirect is set in place. For the last step, a file in the game needs to be changed to work. Somewhere in the game directory, there is a file called openvr api.dll . The OpenComposite repository provides a replacement file that needs to be swapped out in the game files. With this in place, Monado can be booted up, and then the game can start. This should now open the rendered window. The window can now be placed on top of the two displays to see the rendered view in the headset. 74 Chapter A. List of Acronyms VI-SLAM Visual-Inertial SLAM VO Visual Odometry BIBLIOGRAPHY 75 Bibliography [1] M.L. Duarte, L.R. Santos, J.B. Guimar˜aes J´unior, and M.S. Peccin. Learning anatomy by virtual reality and augmented reality. a scope review. Morphologie, 104(347):254–266, 2020. (page 1) [2] I. E. Sutherland. A head-mounted three dimensional display. Unpublished, 1968. (page 3) [3] J. Lanier. Dawn of the New Everything: A Journey Through Virtual Reality. Henry Holt and Company, New York, 2017. (page 3) [4] T. P. Caudell and D. W. Mizell. Augmented reality: an application of heads-up display technology to manual manufacturing processes. In Proc. 25th Hawaii Int. Conf. System Sciences, volume II, pages 659–669, 1992. (page 3) [5] Third generation of video game consoles. https://en.wikipedia.org/w/index. php?title=Third_generation_of_video_game_consoles&oldid=1297671282, June 2025. (page 3) [6] S. Boyer. A virtual failure: Evaluating the success of nintendo’s virtual boy. The Velvet Light Trap, 64, 2009. (page 3) [7] Oculus rift. https://en.wikipedia.org/w/index.php?title=Oculus_Rift& oldid=1295469096, June 2025. (page 4) [8] Vr headsets – worldwide |statista market forecast. http://frontend.xmo.prod. aws.statista.com/outlook/cmo/consumer-electronics/gaming-equipment/ vr-headsets/worldwide. (page 4) [9] Holokit: Open-source ar headset for creative spatial computing. https://holokit. io/. (page 4) [10] Relativty an open-source vr headset for $200. https://www.relativty.com/. (page 4) [11] Sensics & Razer. Osvr: Open source virtual reality. http://www.osvr.com, 2015. Accessed: 2025-07-10. (page 4) [12] Bambu lab |unleash your creativity with bambu lab 3d printers |bambu lab us. https://bambulab.com/en-us/. (page 4) [13] R. Kumar and S. Kumar. Trending applications of 3d printing: A study. Asian Journal of Engineering and Applied Technology, 9(1):1–12, January 2020. (page 4) 76 [14] P. Troxler. Fabrication laboratories (fab labs). In J.-P. Ferdinand, U. Petschow, and S. Dickel, editors, The Decentralized and Networked Future of Value Creation: 3D Printing and its Implications for Society, Industry, and Sustainable Development, pages 109–127. Springer International Publishing, Cham, 2016. (page 4) [15] J. Xiong, E.-L. Hsiang, Z. He, T. Zhan, and S.-T. Wu. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light: Science & Applications, 10(1):216, 2021. (page 5) [16] Z. Ren, X. Fu, K. Dong, Y. Lai, and J. Zhang. Advanced study of optical imaging systems for virtual reality head-mounted displays. Photonics, 10(5):555, 2023. (page 5) [17] K. Bang, Y. Jo, M. Chae, and B. Lee. Lenslet vr: Thin, flat and wide-fov virtual reality display using fresnel lens and lenslet array. IEEE Transactions on Visualization and Computer Graphics, 27(5):2545–2554, 2021. (page 5) [18] B. Narasimhan. Ultra-compact pancake optics based on thineyes super-resolution technology for virtual reality headsets. In Proc. SPIE, page 134, May 2018. (page 5) [19] F. Alam, M. Elsherif, B. AlQattan, A. Salih, S. M. Lee, A. K. Yetisen, S. Park, and H. Butt. 3d printed contact lenses. ACS Biomaterials Science & Engineering, 7(2):794–803, February 2021. (page 5) [20] P. Nair S., H. Wang, J. Trisno, Q. Ruan, S. D. Rezaei, R. E. Simpson, and J. K. W. Yang. 3d printing mesoscale optical components with a low-cost resin printer integrated with a fiber-optic taper. ACS Photonics, 9(6):2024–2031, June 2022. (page 5) [21] Khronos Group. Openxr specification. https://www.khronos.org/registry/ OpenXR/specs/1.0/html/xrspec.html, 2019. Version 1.0; accessed 2025-06-29. (page 5) [22] J. Norlin. Light Field Visualizer: A light field visualizer using OpenGL with OpenXRintegration. Mid Sweden University, 2023. (page 5) [23] C. A. P. de Sousa, H. Hamann, and O. Deussen. Mesh2slam in vr: A fast geometrybased slam framework for rapid prototyping in virtual reality applications. In Proc. IEEE Conf. on Virtual Reality and 3D User Interfaces (VRW), pages 57–62, March 2025. (page ) [24] Steam. Steamvr – ¨ Ubergang zu openxr: Steam-news, June 2020. (page ) [25] I. S. Abeywardena. Oxref: Open xr for education framework. The International Review of Research in Open and Distributed Learning, 24(3):185–206, September 2023. (page 5) BIBLIOGRAPHY 77 [26] Steam. Steam-hardware & software survey. https://store.steampowered.com/ hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam. (page 6) [27] Oculus all in on openxr: Deprecates proprietary apis. https://developers.meta. com/horizon/blog/oculus-all-in-on-openxr-deprecates-proprietary-apis/. (page 6) [28] M. Huzaifa, R. Desai, S. Grayson, X. Jiang, Y. Jing, J. Lee, F. Lu, Y. Pang, J. Ravichandran, F. Sinclair, B. Tian, H. Yuan, J. Zhang, and S. V. Adve. Illixr: Enabling end-to-end extended reality research. In Proc. IEEE Int. Symp. on Workload Characterization (IISWC), pages 24–38, November 2021. (page 6) [29] Collabora. Monado: Open source openxr™runtime. https://monado.dev/, April 2025. Version 25.0.0. (page 6) [30] R. Trimananda, H. Le, H. Cui, and J. T. Ho. Ovrseen: Auditing network traffic and privacy policies in oculus vr. Unpublished, 2025. (page 6) [31] Wikipedia. Oculus Rift — Wikipedia, the free encyclopedia. http://de.wikipedia. org/w/index.php?title=Oculus%20Rift&oldid=251031224, 2025. [Online; accessed 14-July-2025]. (page 6, 16) [32] Yawen Chen, Chenglong Fu, Winnie Suk Wai Leung, and Ling Shi. Drift-free and self-aligned imu-based human gait tracking system with augmented precision and robustness. IEEE Robotics and Automation Letters, 5(3):4671–4678, 2020. (page 6) [33] Angelo Maria Sabatini. Estimating three-dimensional orientation of human body parts by inertial/magnetic sensing. Sensors, 11(2):1489–1525, 2011. (page 6) [34] Gabriele Ligorio, Elena Bergamini, Ilaria Pasciuto, Giuseppe Vannozzi, Aurelio Cappozzo, and Angelo Maria Sabatini. Assessing the performance of sensor fusion methods: Application to magnetic-inertial-based human body tracking. Sensors, 16(2), 2016. (page 6) [35] Suvi Alanko. Comparing Inside-out and Outsidein Tracking in Virtual Reality. (page 7) [36] Ra´ul Mur-Artal and Juan D. Tard´os. ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE Transactions on Robotics, 33(5):1255– 1262, 2017. (page 7) [37] Ra´ul Mur-Artal, J. M. M. Montiel, and Juan D. Tard´os. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5):1147–1163, 2015. (page 7) 78 [38] V. Usenko, N. Demmel, D. Schubert, J. Stueckler, and D. Cremers. Visual-inertial mapping with non-linear factor recovery. IEEE Robotics and Automation Letters (RAL) & Int. Conference on Intelligent Robotics and Automation (ICRA), 5(2):422–429, 2020. (page 7, 37) [39] Varjo. Varjo xr-4 secure edition. (page 10) [40] Unboundxr. Varjo xr-4 secure edition price. (page 10) [41] BigscreenVR. Bigscreenvr. (page 10) [42] Martin Kesselman. Current cite-ings from the popular and trade computing literature: Google cardboard – virtual reality for everyone. Library Hi Tech News, 33(4):15–16, 06 2016. (page 16) [43] V. Andronov. Overview and comparison of pla filaments commercially available. Virtual and Physical Prototyping, 2023. (page 18) [44] Autodesk. Autodesk fusion. https://www.autodesk.com/products/fusion-360/ overview. (page 18) [45] J. Park, Q.-Y. Zhou, and V. Koltun. Colored point cloud registration revisited. In Proc. IEEE Int. Conf. Computer Vision (ICCV), pages 143–152, October 2017. (page 34) [46] F. Steinbr¨ucker, J. Sturm, and D. Cremers. Real-time visual odometry from dense rgb-d images. In ResearchGate, 2011. (page 34) [47] Rgbd odometry – open3d primary (c6d474b) documentation. https://www.open3d. org/docs/latest/tutorial/pipelines/rgbd_odometry.html. (page 34) [48] Nugetforunity. https://github.com/GlitchEnzo/NuGetForUnity. accessed on 2025-08-10. (page 35) [49] Mqttnet. https://www.nuget.org/packages/MQTTnet/. accessed on 2025-08-10. (page 35) [50] Y. Yıldırım, I.S¸ahbaz, T. Kar, G. Ka˘gan, M. Taner, ˙ I. Armagan, and B. Cakici. Evaluation of interpupillary distance in the turkish population. Clinical Ophthalmology, 9:1413–1416, August 2015. (page 36) [51] Virtual display driver. https://github.com/VirtualDrivers/ Virtual-Display-Driver. accessed on 2025-08-10. (page 50) [52] Hannah Fisher. Demystifying Fullscreen Optimizations, December 2019. (page 56)