Head-Worn Augmented Reality for Real-Time Navigation Assistance and Event Forecasting in Maritime Operations

© 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 Head-Worn Augmented Reality for Real-Time Navigation Assistance and Event Forecasting in Maritime Operations 1st Georgios Lamprinakis School of Electrical and Computer Engineering Technical University of Crete Chania, Greece [email protected] 2nd Ioannis Safranoglou School of Electrical and Computer Engineering Technical University of Crete Chania, Greece [email protected] 3rd Georgios Grigoropoulos Senior R&D Manager kpler [email protected] 4th Katerina Mania School of Electrical and Computer Engineering Technical University of Crete Chania, Greece [email protected] Abstract—Human-robot interaction (HRI) in maritime operations is critical for safe and effective collaboration between human operators and autonomous or semi-autonomous vessel systems. In high-risk marine environments, head-worn Augmented Reality (AR) can enhance human-in-the-loop capabilities by delivering real-time navigational data, proactive hazard forecasts, and intuitive interaction frameworks. This paper presents an innovative head-worn AR navigation and event-forecasting system for maritime applications, emphasizing situational awareness, real-time data visualization, and robust calibration across variable ship infrastructures. By leveraging external forecast data and uncertainty visualizations, our approach provides ship captains with critical information about potential collisions and route deviations while maintaining minimal occlusion. We report on two sea trials that informed our design and led to adjustments in the AR “window” placement method for improved alignment and flexibility. Feedback from these trials highlights the potential of comprehensive, real-time AR visualizations to enhance maritime safety and decision-making in complex operational environments. Index Terms—Augmented Reality, Maritime Navigation and Forecasting, Collision Avoidance, HoloLens I. INTRODUCTION Head-worn AR interfaces are transforming maritime operations by enhancing situational awareness, as shown in a maritime eye-tracking study for real-time decision-making [1]. AR is deployed for navigation, providing real-time route and hazard visualization as well as for training [2], through realistic scenario simulations [3] including maintenance [4] and repairs [5], enabling on-shore experts to assist crew members. Early Maritime AR Systems (MARS) aimed to unify ship instrument data in a single AR interface. Even the initial prototypes improved situational awareness and reduced cognitive load [6]. Others combine AIS, radar, and route data through hybrid interfaces that merge world-referenced 3D visuals with contextual and non-contextual information [7]. AR is not only used in open-ocean navigation but also to assist captains during harbor maneuvers, where smart glasses provide critical information for precise navigation and improve situational awareness in confined environments [8]. While most early maritime AR systems relied on monitors to provide AR content [7], [9], [10], recent advancements have shifted towards the integration of headsets and smart glasses [11], [12], keeping information directly in operator’s line of sight, enabling fully hands-free [13]. However, enabling interactive AR visualizations of real-time data (e.g. forecast hazards with uncertainty indicators) without occluding the operator’s view remains an open challenge [14]. Presenting critical information via head-worn AR, while avoiding new visual obstructions, is essential for maintaining safety and situational awareness [15]. To address these challenges, we introduce an innovative head-worn AR navigation system developed to enhance wayfinding and situational awareness for ship captains, while at sea. It features a range of interaction techniques and includes a mini-map that highlights key points of interest (POIs) and uncertainty cones for projected navigational paths, offering a comprehensive overview of the ship’s surroundings. The specific contributions of this paper include: •Real-Time AR Overlay System: An AR system that integrates a server with head-worn AR devices to dynamically overlay safe routes, ship trajectories, and hazard zones directly into the captain’s field of view, enhancing situational awareness and navigation efficiency. •Integration of GPS and AIS Data: Seamless incorporation of GPS and Automatic Identification System (AIS) data with AR to visualize critical POIs (e.g., ports and restricted zones) on the horizon, as well as hazards near the ship. •Uncertainty Visualization and Interaction: Uncertainty © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 visualization with gazeand gesture-based interactions for predicting ship trajectories over time. Visual information is tailored to each vessel’s structure, with toggle controls to allow unobstructed viewing. •Proof-of-Concept in Simulation: Proof-of-concept evaluations in simulated maritime settings to validate system functionality and usability before real-world deployment. •Expert User Evaluation: Comprehensive evaluation by expert users in collision avoidance scenarios, demonstrating the system’s potential to enhance navigation and decision-making in high-stakes maritime operations. II. RELATED WORK A. AR for Maritime Navigation AR systems for maritime navigation have been explored in both commercial and academic contexts. The Furuno Envision System [16] is a commercial monoscopic AR solution that overlays navigational data (e.g., AIS information, routes, waypoints) onto a camera feed displayed on a bridge-mounted monitor. Although it enhances situational awareness, it fails to reduce head-down time, calling its effectiveness into question [1]. Moreover, no operational deployment or technology readiness level has been reported, casting doubt on its practical utility [17]. Our AR system not only visualizes navigational data but also introduces interactive event forecasting with novel uncertainty visualization for prediction accuracy and real-time navigation assistance. Oh et al. [18] presented a monoscopic AR prototype that combined 2D and 3D graphics to display heading, route, and traffic information. Validated in real-world ocean settings, it faced usability challenges such as information overload and overlapping visuals, which reduced clarity and raised concerns about workload. Similarly, Lee et al. [7] developed a computer vision-based system for vessel detection, using 2D AR overlays for AIS and ship data; however, the lack of a structured integration framework limited its scalability in real navigation scenarios. Leite et al. [19] focused on obstacle detection and visualization, utilizing grid overlays and zoomed-in views to enhance situational awareness. Our system focuses on enhancing usability without overloading the user’s field of view, providing intuitive gaze-based interaction and interactive toggle control related to information visualization by the captain. Frydenberg et al. [11] explored stereoscopic AR concepts for maritime navigation, testing prototypes on the HoloLens 1 as part of the SEDNA project, which focused on Arctic navigation under extreme conditions. Their findings were later formalized into interface design guidelines by Nordby et al. [20], establishing a foundation for developing stereoscopic AR systems with defined application components and display zones. Frydenberg et al. further developed an AR interface for Arctic icebreaker operations, demonstrating improved crew communication and situational awareness in convoy navigation [21]. Their field evaluations underline AR’s value for human–robot teamwork under extreme conditions. Building on this foundation, the proposed system introduces a novel stereoscopic AR interface, interactively placed in any ship according to its structure to match, for instance, bridge window shape, dynamically visualizing safe routes, highlighting critical hazards, and addressing positional uncertainties in real time, advancing AR navigation technology for improved decision-making and safety. B. Uncertainty Visualization Visualizing uncertainty helps users understand data reliability and variability, aiding informed decision making [22]. Despite its importance, uncertainty visualization in maritime contexts remains under-explored. Plumejeaud-Perreau and Marzagalli [23] developed a tool for visualizing uncertainty in historical maritime routes using color-coded maps and schematic representations. Riveiro et al. [24] examined uncertainty visualization in an air-defense target identification scenario using semi-transparent circles and line thickness to convey positional uncertainty and sensor accuracy. Operators with access to these visualizations were more cautious and effective in prioritizing threats. Broad et al. [25] examined the public misinterpretation of hurricane path uncertainty visualizations, such as the widely used ”cone of uncertainty.” Their findings emphasize the necessity of clear and intuitive designs to avoid ambiguity and misinterpretation. Padilla et al. [26] demonstrated the effectiveness of ensemble displays and Hypothetical Outcome Plots in improving risk assessment by presenting potential variations clearly, applicable to predicting ship trajectories and hazard zones. Liu et al. [27] proposed a method for visualizing time-specific hurricane predictions using storm path ensembles. Their approach incorporated radial basis functions and simplicial depth to estimate uncertainty, using overlapping confidence intervals to intuitively represent positional risk, offering techniques applicable to maritime navigation. The proposed AR system incorporates an uncertainty cone to project navigational inaccuracies along predicted routes. The cone’s gradient dynamically adjusts to reflect varying levels of GPS accuracy and environmental uncertainty. By leveraging concepts from existing studies, such as color-coded representations and semi-transparent overlays, the system aims to ensure clarity and reduction in cognitive load. Moreover, the novel integration of uncertainty visualization into the AR interface promotes cautious navigation, enhancing safety and situational awareness in high-stakes maritime environments. III. SYSTEM OVERVIEW Our system architecture for AR-based maritime navigation utilizes head-worn AR to provide real-time data visualization for enhanced situational awareness and decision-making. By integrating live navigational and environmental data, the system dynamically adjusts visual components, aiding ship captains in high-pressure maritime scenarios. The following system features were developed based on feedback from ship captains during real-world testing. HoloLens 2 AR Headset: Enables hands-free operation and immersive visualization of maritime scenarios, e.g. routes, Points of Interest (POIs), and hazard zones. Data Integration: © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 Processes navigational data, e.g. AIS and GPS inputs [28], stored in a JSON format or streamed via Kafka topics for real-time updates. Intermediate Server: Facilitates communication between data sources and the AR device, ensuring smooth data flow for dynamic visualization. Dynamic Route and Hazard Visualization: Displays real-time updates of ship routes, collision predictions, and hazard zones, including uncertainty cones for projected trajectories [29]. HandsFree Operation: Gaze and gesture-based interaction allow captains to access critical information without losing focus on navigation tasks. Real-Time Data Updates: Ensures up-todate visualization of ship trajectories, hazard zones, and POIs, for rapid decision-making. POIs and Mini-Map Integration: Displays key points (e.g., other ships, rocks, lighthouses) on the horizon while a mini-map offers an overview of the navigational environment. Route and Hazard Visualization: Projects navigational routes onto the water surface below the horizon for enhanced depth perception. Uncertainty cones relate to potential trajectory deviations. IV. IMPLEMENTATION A. Data For maritime navigation, our AR system integrates contextual real-time data to accurately visualize routes, collision points, and ship trajectories. Live AIS data is analyzed to predict trajectories and identify potential collision points, and a custom onboard phone application continuously transmits ship-specific parameters (speed, heading, position), ensuring real-time updates for dynamic AR overlays. Ship-Specific Data consists of the vessel’s unique identifier (MMSI), real-time speed, and courses are streamed via the phone app, reflecting current behavior and ensuring AR visualizations align accurately with the ship. AIS Data Analysis provides real-time AIS-based positional tracking and trajectory predictions. The system uses incoming AIS data (positions, speed, course) to predict future ship routes and highlight potential collision points, enabling timely and accurate visual cues. Spatial Geo-References anchor key virtual elements (routes, collision points, ship icons) are anchored to geospatial coordinates. Waypoints defined by latitude/longitude are projected onto the AR scene (e.g., on the water surface) in alignment with the real world. Collision points are marked at specific coordinates and update dynamically based on AIS data and predictions, helping the captain anticipate risks. By leveraging ship-specific input and uncertainty-aware visualization, the system delivers an accurate representation of maritime routes and hazards. Captains have access to reliable navigation data, both in AR and on the integrated mini-map, enabling improved situational awareness and robust real-time decision-making during high-stakes maritime operations. B. Server Architecture and Data Handling The system utilizes a robust server infrastructure and realtime data integration to deliver dynamic visualizations directly to ship operators as they navigate the vessel. At the core of the back-end architecture is a Dockerized Kafka server and an Intermediate Node Server (Figure 1). Kafka’s primary capabilities include publishing and subscribing to streams of records, storing these records in their generated sequence, and processing them in real time. Employing a publish-subscribe model, Kafka allows producers to write data to topics, while consumers read from these topics, enabling scalable and resilient data transmission. By initializing Kafka server containers with Zookeeper and Kafka images, the system ensures reliable message brokering and processing. To facilitate data flow, an Intermediate Node Server was developed using Node.js, selected for its event-driven asynchronous architecture. The Node server connects the Unity application to Kafka, utilizing libraries like KafkaJS for Kafka client operations and ws for WebSocket communication. Data transmission occurs in two main phases: from Kafka topics to the Node server, and from the Node server to the HoloLens 2. Using KafkaJS, the Node server subscribes to forecast topics, processes each message, and relays the essential data to the HoloLens 2 over WebSockets. This pipeline keeps the headset updated in real time, where the AR app injects the information directly into the visual layer. Because HoloLens 2 has no built-in GPS, a companion Android app streams live coordinates to the same Node server, which forwards them to the device to maintain precise alignment of virtual POIs and routes with the ship’s true position. Fig. 1. System architecture C. AR Interaction Our system based on the HoloLens 2 device deploys hands-free operation, visualizing real-time navigational data, including predicted ship trajectories, safe routes, hazard zones, and uncertainty cones indicating positional deviations. It also visualizes key Points of Interest (POIs) and integrates a minimap for enhanced situational awareness. While the HoloLens 2 provides significant advantages, its display performance in bright outdoor environments poses challenges. The device’s display becomes less visible in direct sunlight, and tracking can be unstable. Bright lighting often causes camera over-saturation, while low-light conditions fail to capture sufficient detail, affecting overall system performance. For optimal tracking performance it is ideal to control lighting within a range of 500-1000 lux. To address these limitations, we designed and implemented custom ”sunglasses” for the HoloLens 2 in the form of optical filters [30]. These optical © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 filters improve visibility in bright environments, allowing the device to function effectively even in direct sunlight. 1) Hand Menu and User Interface: The user interface (UI) of our AR system is designed to be intuitive and easily accessible through hand gestures. The main menu (Figure 2(a)) is accessed by facing one of the palms towards the user, utilizing the hand tracking feature of the HoloLens 2. The menu is structured in layers, providing access to calibration buttons within its layered architecture (Figure 2(b)). The toggles for Points of Interest (POIs), Route, and Map are located in two panels positioned on the upper side of the ship bridge, offering the captain quick and convenient access to these essential controls. During the calibration process, the menu adapts to display only the necessary buttons, ensuring a streamlined and focused user experience by removing non-essential options. (a) Hand menu interface (b) Second layer Fig. 2. Multi-layered hand menu 2) Calibration Process: Accurate calibration is crucial, ensuring that virtual elements align correctly with the real world. Calibration involves four stages: Initial Calibration: Orientation to True North: This ensures that the AR system’s 3D scene is correctly aligned with the real-world. The user faces true north using a compass and then presses a button on the HoloLens to accordingly orient the system. This typically results in a small offset of about 4-5 degrees. While this offset may introduce minor inaccuracies, it does not significantly misalign the placement of POIs, ensuring reliable AR functionality. Upon startup, the user follows on-screen instructions to face true north and presses the calibration button on the HoloLens to lock the orientation. If alignment is lost or adjustments are needed, calibration is repeated anytime through the main menu. Horizon Alignment: After completing the initial orientation, the user is guided to align the system with the perceived horizon. This step ensures that virtual elements are accurately placed relative to the horizon. A new menu screen shows a line parallel to the ground. The user adjusts this line to match the realworld horizon and presses the ”Align” button to confirm. Recalibration can be revisited via the main menu if the horizon alignment requires fine-tuning allowing for precise placement of virtual objects above or below the horizon, enhancing depth perception and realism. Virtual Windows Setup: This stage of calibration focuses on creating and positioning virtual windows that user anchors to ship’s physical windows constraining AR content to those areas. Utilizing the updated calibration method, the user interacts with a gaze-following sphere that navigates the AR environment, automatically colliding with the ship’s physical walls to assist in accurate placement. Fig. 3. POI’s alignment process menu Corner Placement: A small sphere appears at the user’s gaze intersection point with the ship’s structure. The user positions this sphere at the desired corner of an AR window by performing a pinch-and-hold gesture for two seconds. During this gesture, a circular slider appears around the sphere, gradually filling as time passes. Once the slider is completely filled after two seconds, the corner placement is confirmed, and the sphere’s position is locked. If the user releases the pinch before the slider is fully filled, the timer resets, and the corner placement is not confirmed. Verification: Upon successfully pinching and confirming the corner placement, a colored square is created at the chosen point, providing visual feedback to the user and ensuring accurate placement. Repeat for All Corners: The user repeats the pinch-and-hold gesture to position the remaining three corners of the window. This method allows for the creation of non-parallelogram rectangular windows, offering greater flexibility in alignment with the ship’s actual window shapes. Multiple Windows Creation: Users create multiple AR windows by repeating the corner placement process, enabling tailored visualization across varied sections of the ship’s bridge. Deletion Capability: Each AR window includes a delete button centrally located within the window. Users can easily remove unwanted windows by selecting this button, facilitating quick adjustments without cluttering the interface. (a) Placing AR window’s corner (b) Newly created AR window Fig. 4. AR window alignment process All AR windows are initially colored to indicate their pending fixation. Once all corners of a window are confirmed, the window transitions to a transparent state, ensuring an unobstructed view for the user while maintaining the integrity © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 of the AR overlays. This method not only accelerates the calibration process but also enhances accuracy by allowing precise alignment based on the ship’s actual window geometry. Additional calibration procedure is implemented using QR codes strategically placed around the ship’s bridge. This QR code-assisted calibration helps ensure that the AR elements are properly aligned as the ship moves. This procedure guarantees that the positioning of virtual elements remains accurate and consistent with the ship’s movement, further enhancing the overall AR experience. The procedure is as follows: QR Code Placement: Locate all QR codes installed around the ship’s bridge. Calibration Action: Each QR code features a calibrate button on its top; press the button to initiate calibration for that specific location. Final Calibration: Once all QR code buttons have been pressed, open the hand menu and select ”Calibrate QR Codes” to complete the calibration process. Map and Assistant Panel Placement: In this stage, the system creates four panels and a map. Suggested positions include placing the map centrally above the bridge window, toggle panels next to it and ShipInfo panels (Figure 7(b)) on the window pillars. This setup ensures easy access and avoids obstructing captain’s view. 3) AR Windows : AR Windows provide a seamless way to create virtual portals into the AR environment (Figure 5), allowing users to view specific data or objects without cluttering their workspace. Leveraging the updated calibration method, AR Windows are intuitively placed and resized to fit the user’s needs, offering flexibility and enhanced situational awareness in real-time navigation scenarios. (a) From this angle, specific virtual items are visible within the portal’s frame (b) Adjusting the perspective exposes different parts of the anchored content Fig. 5. Demonstration of AR window functionality 4) Points of Interest (POIs): Our AR system provides real-time 3D representations of key locations (POIs) such as collision points, other ships, reefs, and others. This feature leverages GPS coordinates to accurately place POIs within the Unity environment. POIs are visualized exclusively through AR Windows, which serve as virtual windows into the AR environment, ensuring a clean and unobstructed workspace. Most of the symbols used for the POIs are directly inspired by those found in official nautical charts, including those used in widely adopted platforms like Navionics. This design choice ensures consistency with established professional maritime navigation standards, enabling captains to intuitively recognize and interpret the visualized elements. By aligning with familiar symbols, the system significantly reduces the training time required for captains to effectively use the AR system(Figure IV-C4). Static POIs are pre-loaded locations such as reefs or other objects that remain unchanged. Dynamic POIs are real-time updates for locations such as other ship positions or significant points occurring in real-time, dynamically acquired and broadcast through the Kafka-Node server pipeline. To accurately visualize GPS locations in AR relative to the user’s position, the real-world latitude and longitude coordinates must be converted into AR-compatible values, involving calculating the differences in latitude and longitude between the user’s position and the target location. The latitude difference (∆lat) is obtained by subtracting the user’s latitude from the target latitude and then scaling the result by approximately 111,000, representing the approximate length in meters of one degree of latitude. For the longitude difference (∆lon), the process is slightly more complex due to the Earth’s curvature. It requires subtracting the user’s longitude from the target longitude and multiplying the result by the length of one degree of longitude at the user’s latitude. This length is approximated as 111,000 multiplied by the cosine of the user’s latitude expressed in radians. By performing these conversions, GPS coordinates can be correctly represented in the AR space with scaling and alignment to the user’s perspective. (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Fig. 6. a) Collision point b) Collision ship c) Ship passing by d) Arrow at ships’ route point e) Arrow at other ships route point f) Red lighthouse g) Green lighthouse h)Reef i) Wreck j) Unknown danger k) Mooring buoy l) Special Purpose or General buoy Due to typical GPS accuracy limitations, the placement of these points can have an offset of around 3-5 meters under optimal conditions. This degree of accuracy is sufficient for most systems. POIs are designed to be gaze-interactable. When the user focuses on a POI for a short period, an information panel opens, revealing more details about that POI, as shown in Figure 7(a). This interaction method provides the captain with critical information quickly, without manual input, maintaining hands-free operation. 5) Map: The mini-map, created using the MapBox API, provides real-time 2D navigation support by dynamically displaying the ship’s location, routes, and Points of Interest (POIs) based on GPS and AIS data processed through our custom server. To address GPS inaccuracies, a circle is displayed around both our ship and the opposing ship, representing © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 potential positional deviations, a method commonly used in platforms like Google Maps. (a) Info for collision ship POI (b) Information Fig. 7. User interfaces For collision scenarios, the opposing ship’s route is visualized with a line accompanied by a cone of uncertainty inspired by hurricane path prediction models. The uncertainty cone widens as the ship progresses, reflecting increasing positional uncertainty over time. Similarly, our ship’s path is represented by a line with an adjacent uncertainty cone, accounting for potential deviations due to navigational adjustments, external forces such as currents, or uncertainties in GPS data. This combined visualization (Figure 8) ensures that captains have a realistic understanding of safe zones and the flexibility within their route. The mini-map is strategically placed just above the bridge windows (Figure 10), ensuring it remains accessible without obstructing the captain’s view of the sea. Captains can also adjust the map’s position for better orientation. By combining real-time data, uncertainty visualization, and ergonomic placement, the mini-map enhances situational awareness and supports safer navigation. Fig. 8. Map with Routes, Uncertainty Routes, Danger Area, POIs and a red line indicating the Heading. Photo from second sea trial 6) Route Visualization: The system projects the planned route onto the water surface below the horizon line, which improves depth perception and compensates for HoloLens 2 limitations with distant object rendering. The route is viewed exclusively through the AR Window, being integrated seamlessly into AR without obstructing critical instruments. The route itself is composed of sequential points received from our custom-built server, dynamically integrating realtime GPS and AIS data to ensure accuracy and relevance to the ship’s current position and destination. These points define the proposed safe path to follow, factoring in hazards and restricted zones. Each point is oriented toward the next, forming a continuous path that visually guides the captain in the correct direction. Despite the real-time data integration, GPS accuracy limitations introduce a degree of positional uncertainty, typically around 3-5 meters under optimal conditions. To mitigate the impact of potential inaccuracies, the system incorporates a cone of uncertainty (Figure 9) along the visualized route. Instead of a single thin line, the route appears with a widening cone that extends outward from the ship’s current position, indicating the area in which the ship could realistically be, considering potential drift or errors. This approach provides a buffer zone visualization: even if the exact future path is uncertain, it shows a safe corridor. It is inspired by techniques in hurricane track forecasting where widening cones indicate uncertainty further out in time. Incorporating this uncertainty visualization ensures the captain remains aware of possible deviations and is not overconfident in a single precise track, thereby enhancing safety. Fig. 9. Uncertainty route in the lab V. EVALUATION TRIALS A. Sea Trial on a 44-Meter Cruise Ship The system was initially tested onboard a 44-meter cruise ship operating in an area of heavy maritime traffic and dynamic navigation conditions. Although the vessel remained stationary during the trial, the setting accurately reflected high-traffic maritime scenarios, enabling a realistic preliminary evaluation. Old Window Placement Method. For this first trial, we employed an earlier version of the window placement process. During calibration (or recalibration), the user could create virtual windows that were subsequently adjusted by scaling or tilting via a set of incremental controls accessible from a small menu above each window. Specifically, the user could scale x up,scale x down,scale y up, and scale y down, along with some limited rotation and tilt operations. While this method allowed basic manipulation of window size and orientation, it was constrained by only handling near-rectangular shapes. Users had to perform multiple, often tedious, adjustments to approximate the physical contours of each ship window. © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 Captain’s Feedback. During the 44-meter ship trial, the captain tested several AR system features—route visualization, collision prediction, and uncertainty cones—using speed, heading, and position data streamed in real time(Figure 10). He found the general concept of overlaying navigational data in AR compelling, noting improvements in situational awareness. However, he deemed the existing window placement method cumbersome for aligning AR overlays with the bridge’s actual window geometry. He requested additional Points of Interest (POIs) and more detailed information on each POI. Consequently, toggle controls were introduced for POIs and map elements, placed above the bridge to mitigate visual clutter. At this stage, we had not yet implemented a dedicated toggle for showing or hiding the route’s elements or POIs, but discussions began about adding one in future iterations. We noted persistent glare problems at certain angles—particularly during dusk or when sunlight directly hit the bridge window—this rose the need for the custom sunglasses that were implemented and tested in the next trial. Additionally, connecting the HoloLens 2 to both a mobile device and a server diminished confidence in overall reliability. Depth perception remained a challenge, as accurately gauging POI distances proved difficult. Despite these concerns, the captain concluded that real-time hazard alerts and route visualization significantly enhanced situational awareness. Fig. 10. Photos from first trial B. Tug Boat Trial on a 30-Meter Vessel Following the cruise ship test, a second trial was conducted on a 30-meter tug boat to gather further performance data and user feedback. The vessel again remained stationary, but the smaller bridge layout, numerous windows, and high ambient light conditions posed new challenges. Streamlined Window Placement. In this trial, our new window placement method proved easy to operate, minimized setup time, and seamlessly handled trapezoid-shaped windows. Its adaptability and speed significantly reduced user frustration (Figure 11). Taking into account the glare issues as identified in the first trial, we deployed custom “sunglasses” for the HoloLens 2 in this second test, offering a tinted filter that mitigated some of the direct sunlight. Although it provided moderate relief, the bright conditions aboard the tug boat still caused partial washout of AR elements. The captain on the 30-meter tug boat acknowledged that several improvements had been made since the first trial: New Window Placement: Easier to operate and with many more capabilities than before. Route/POIs Toggles: The captain appreciated being able to toggle the route elements or POIs on or off, reducing visual overload during certain maneuvers or navigational scenarios. New POIs and Detailed Info Panels: An expanded catalog of POIs was introduced, and the captain could now select each POI to access more in-depth navigational or contextual information. Custom Sunglasses: Glare was reduced to some extent, allowing for better visibility of AR overlays in bright environments. The captain also noted that route visualization remained difficult to interpret due to limited depth cues, even though toggling it off helped reduce clutter. Collectively, these issues underscored the need for a more robust and flexible window placement system, one capable of handling non-standard window shapes and simplifying the calibration process for new users. Despite its limitations, the tug boat trial further validated the potential of the system to improve maritime navigation. The captain recognized improvements in POI management, calibration speed, and route toggling but emphasized that display visibility, precise window alignment, and depth perception are critical pain points that must be resolved for full operational viability. These findings reinforced the design shift toward a more adaptive calibration workflow—one that is easier to operate, accommodates varying window geometries, and minimizes user frustration. Crucially, discussions with the tug boat captain about handling non-parallelogram windows led to the conception of our revised window placement approach. Rather than relying on incremental menu controls, the new method utilizes gaze-following spheres and pinch-and-hold gestures with a circular slider to rapidly define each window corner. This approach inherently supports arbitrary window shapes, drastically reducing the time and effort required to accurately align AR overlays. Fig. 11. Photos from second trial VI. CONCLUSION Both trials confirm that an AR-based navigation system can improve situational awareness, facilitate route planning, and assist in collision avoidance. The first trial revealed © 2025 IEEE. This is the author’s pre-print version of the article published in the proceedings of the 2025 Symposium on Maritime Informatics and Robotics (MARIS) The final version of this record is available at: https://doi.org/10.1109/MARIS64137.2025.11139568 fundamental issues related to window placement, the breadth of POIs, and connectivity overhead, while the second trial reinforced the need for better glare management, a simplified calibration process, and improved depth cues. The captain of the tug boat explicitly acknowledged progress in user interface design, affirming that the modifications inspired by feedback from the 44-meter ship had already made the system more accessible. These phased evaluations on both a 44meter cruise ship and a 30-meter tug boat demonstrated the system’s support for safe and efficient maritime navigation. Key features such as route visualization, collision prediction, and real-time hazard indicators were beneficial, although each trial highlighted technical and ergonomic areas that needed further attention. Visibility challenges in varying lighting conditions, depth perception difficulties, and the need for a streamlined connectivity set-up remain concerns. With each set of trials, the system evolved to address user feedback. Future work will focus on improving display technologies to combat glare, refining the calibration workflow for new AR users, enhancing spatial cues to improve depth perception, and optimizing ergonomics to reduce physical strain. By resolving these challenges, the AR-based maritime navigation system is poised to become a trusted, indispensable tool in modern maritime operations, ultimately contributing to safer and more efficient navigation practices. Acknowledgment This work was supported by the EU project CREXDATA under Horizon Europe agreement No. 101092749. REFERENCES [1] R. O. Odd Sveinung Hareide, “Maritime usability study by analysing eye tracking data,” in Journal of Navigation · April 2017, 2017. [2] C. W. M. N. C. W. O. Muhammad Raziq Kazura, Mohammed Ismail Russtam Suhrab1, “Enhancing maritime education and training (met) through virtual andaugmented reality: A voyage into immersive learning,” in JOURNAL OF MARITIME RESEARCH, 2024, pp. 339–348. [3] S. K. R. Steven C. Mallam, Salman Nair, “Rethinking maritime education, training, and operations in the digital era: Applications for emerging immersive technologies,” in J. Mar. Sci. Eng. 2019, 2019. [4] W. C. (2016), “W¨ artsil¨ as service ensures a reliable and pleasant cruise for passengers of m/s artania,” in W¨ artsil¨ a Corporation, 2016. [Online]. Available: https://www.wartsila.com/media/news/ 02-11-2016-wartsilas-service-ensures-a-reliable-and-pleasant-cruise-for-passengers-of-ms-artania [5] W. C. (2018), “Augmented reality creates a new dimension in marine maintenance services,” in W¨ artsil¨ a Corporation, 2018. [Online]. Available: https://www.wartsila.com/media/news/ 31-07-2018-augmented-reality-creates-a-new-dimension-in-marine-maintenance-services [6] H. S. Y. A. Teo Chee Hong and C. W. L. Kenny, “Assessing the situation awareness of operators using maritime augmented reality system (mars),” in Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015, 2015, pp. 1722–1726. [7] K. H. L. Y. W. Jung Min Lee, Byeongwook Nam, “Study on image-based ship detection for ar navigation,” in 2016 6th International Conference on IT Convergence and Security (ICITCS), 2016. [8] A. L. Marie-Christin Ostendorp, Jan Charles Lenk, “Smart glasses to support maritime pilots in harbor maneuvers,” in 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015, 2015. [9] R. S. K. M. Tadatsugi Okazaki, Rei Takaseki, “Development of sea route display system by using augmented reality,” in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017. [10] B.-W. Nam, K.-H. Lee, and J.-M. Lee, “A study on developing image processing for smart traffic supporting system based on ar,” 04 2017. [11] N. K. Frydenberg, S. and J. O. Eikenes, “Exploring designs of augmented reality systems for ship bridges in arctic waters,” in RINA, Human Factors 2018, 2018. [12] V. d. L. Gerald Moulis, “How augmented reality can be fitted to satisfy maritime domain needs – the case of visiprot® demonstrator,” in VRIC ’15: Proc. 2015 Virtual Reality International Conference, 2015. [13] Y. Baashar, G. Alkawsi, W. N. Wan Ahmad, M. A. Alomari, H. Alhussian, and S. K. Tiong, “Towards wearable augmented reality in healthcare: A comparative survey and analysis of headmounted displays,” International Journal of Environmental Research and Public Health, vol. 20, no. 5, 2023. [Online]. Available: https://www.mdpi.com/1660-4601/20/5/3940 [14] I. Safranoglou, A. Stavroulakis, M. Ebel, J. Pottebaum, G. Lamprinakis, D. Dimelli, and K. Mania, “Augmented reality for real-time decisionmaking in flood emergencies,” in 2024 IEEE Intl. Symp. on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2024, pp. 110–116. [15] G. Koulieris, K. Aks¸it, M. Stengel, R. Mantiuk, K. Mania, and C. Richardt, “Near-eye display and tracking technologies for virtual and augmented reality,” Computer Graphics Forum, vol. 38, pp. 493–519, 2019. [16] F. P. S. (2019, “Driving the digitalization of navigation: Envision,” in Furuno Corporation, 2019. [Online]. Available: https://www.furuno. com/special/en/envision/ [17] M. Grabowski, “Research on wearable, immersive augmented reality (wiar) adoption in maritime navigation,” in THE JOURNAL OF NAVIGATION (2015), 2015, pp. 453–464. [18] O.-S. K. Jaeyong OH, Sekil PARK, “Advanced navigation aids system based on augmented reality,” in International Journal of e-Navigation and Maritime Economy 5 (2016) 021 – 031, 2016. [19] B. G. Leite, H. T. Sinohara, N. Maruyama, and E. A. Tannuri, “Maritime navigational assistance by visual augmentation,” Journal of Navigation, vol. 75, no. 1, p. 57–75, 2022. [20] S. F. J. O. E. Kjetil Nordby, Etienne Gernez, “Augmenting openbridge: An open user interface architecture for augmented reality applications on ship bridges,” in 19th Conference on Computer and IT Applications in the Maritime Industries - COMPIT 2020, 2020. [21] S. Frydenberg, K. Aylward, K. Nordby, and J. O. H. Eikenes, “Development of an augmented reality concept for icebreaker assistance and convoy operations,” Journal of Marine Science and Engineering, vol. 9, no. 9, 2021. [Online]. Available: https: //www.mdpi.com/2077-1312/9/9/996 [22] K. Potter, P. Rosen, and C. Johnson, “From quantification to visualization: A taxonomy of uncertainty visualization approaches,” IFIP advances in information and communication tech, vol. 377, 01 2012. [23] S. M. Christine Plumejeaud-Perreau, “Mapping the uncertainty of past maritime routes,” in European Cartographic Conference – EuroCarto 2022, 19–21 September 2022, TU Wien, Vienna, Austria, 2022. [24] M. Riveiro, T. Helldin, G. Falkman, and M. Lebram, “Effects of visualizing uncertainty on decision-making in a target identification scenario,” Computers Graphics, vol. 41, pp. 84–98, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0097849314000302 [25] J. W. Kenneth Broad, Anthony Leiserowitz and M. Steketee, “Misinterpretations of the “cone of uncertainty” in florida during the 2004 hurricane season,” in 2007 American Meteorological Society, 2007, p. 651–668. [26] L. Padilla, M. Kay, and J. Hullman, Uncertainty Visualization, 02 2021, pp. 1–18. [27] L. Liu, M. Mirzangar, R. Kirby, R. Whitaker, and D. House, “Visualizing time-specific hurricane predictions, with uncertainty, from storm path ensembles,” Computer Graphics Forum, vol. 34, 06 2015. [28] I. Chamatidis, G. Spiliopoulos, M. Kaliorakis, G. Grigoropoulos, and K. Bereta, “Synthetic ais dataset of vessel proximity events,” Oct. 2023. [Online]. Available: https://doi.org/10.5281/zenodo.8358665 [29] G. Grigoropoulos, G. Spiliopoulos, I. Chamatidis, M. Kaliorakis, A. Troupiotis-Kapeliaris, M. Vodas, E. Filippou, E. Chondrodima, N. Pelekis, Y. Theodoridis, D. Zissis, and K. Bereta, “A scalable system for maritime route and event forecasting,” in Proceedings of the 27th International Conference on Extending Database Technology (EDBT). New York, NY, USA: Association for Computing Machinery, 2024. [30] J. O. H. Eikenes, “How to create your own hololens sun screen,” in Ocean Industries Concept Lab, 2019. [Online]. Available: https://medium.com/ocean-industries-concept-lab/ how-to-create-your-own-hololens-sun-screen-68c466071a01