Dynamic Property Showcases: Path Planning for Virtual Tours with Neural Radiance Fields

UNIVERSITAT POLITÈCNICA DE CATALUNYA MASTERS THESIS Dynamic Property Showcases: Path Planning for Virtual Tours with Neural Radiance Fields Author: Javier BENITEZ MARIN A thesis submitted in fulfillment of the requirements for the degree of Master’s degree in Advanced Telecommunication Technologies in the Escola Tècnica Superior d’Enginyeria de Telecomunicació de Barcelona Supervisor: Javier RUIZ HIDALGO October 22, 2024 iii Declaration of Authorship I, Javier BENITEZ MARIN, declare that this thesis titled, “Dynamic Property Showcases: Path Planning for Virtual Tours with Neural Radiance Fields” and the work presented in it are my own. I confirm that: • This work was done wholly or mainly while in candidature for a research degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. • I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself. v UNIVERSITAT POLITÈCNICA DE CATALUNYA Abstract Universitat Politècnica de Catalunya Escola Tècnica Superior d’Enginyeria de Telecomunicació de Barcelona Master’s degree in Advanced Telecommunication Technologies Dynamic Property Showcases: Path Planning for Virtual Tours with Neural Radiance Fields by Javier BENITEZ MARIN This project presents a novel approach to virtual camera path planning and exploration in 3D environments, leveraging the capabilities of Neural Radiance Fields (NeRF) for scene representation and novel view synthesis. The objective is to generate optimized camera paths through a pre-trained NeRF model to enable efficient and visually comprehensive virtual tours, particularly in applications like property showcasing. The methodology combines classical and probabilistic path planning techniques, employing Rapidly-exploring Random Trees (RRT) to compute collision-free paths between predefined Points of Interest (POIs) within the NeRF-generated environment. Additionally, the camera orientation at each waypoint is optimized to maximize scene coverage, ensuring that the rendered video provides detailed views of the most significant areas. This work identifies a gap in current research, which predominantly focuses on NeRF-based exploration for real-world robotic navigation, leaving virtual environments unexplored. The project contributes a novel system that adapts NeRF’s rendering capabilities to virtual navigation, emphasizing the optimization of camera paths for high-quality scene visualization without the constraints of physical navigation. The system’s design, path planning, and camera direction optimization methods are detailed, and potential applications of this technology are discussed in the context of virtual tours. vii Acknowledgements I would like to express my deepest gratitude to my project advisor, Javier Ruiz, whose invaluable guidance and insights were instrumental throughout the development of this project. Our weekly discussions were crucial in overcoming the major challenges that arose, and his expertise provided me with the direction I needed to navigate these difficulties successfully. His support and mentorship have been a key factor in the completion of this work. I would also like to thank my family. My sister, Carolina, has always been a role model to follow, inspiring me with her dedication and resilience. To my parents, thank you for your unwavering support and encouragement in everything I do. Your belief in me has been a constant source of motivation. Finally, I extend my appreciation to my friends, who have been by my side throughout this journey. During the writing process, their company and collaboration helped me stay focused, and the shared moments of hard work and breaks made the experience much more enjoyable. Thank you all for your help and support. ix Contents Declaration of Authorship iii Acknowledgements vii 1 Introduction 1 1.1 Objectives ................................... 1 2 State of the Art 3 2.1 Neural Radiance Fields (NeRF) . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 PathPlanning ................................. 5 2.2.1 Classical Path Planning Algorithms . . . . . . . . . . . . . . . . 5 2.2.2 Probabilistic Path Planning Algorithms . . . . . . . . . . . . . . 5 2.3 Exploration and Path Planning in Neural Radiance Fields . . . . . . . . 6 2.4 Identified Gaps in the Research . . . . . . . . . . . . . . . . . . . . . . . 7 3 Methodology 9 3.1 High Level System Overview . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2 DataPreparation ............................... 10 3.3 Occupancy Grid Construction . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3.1 Grid Overlay and Voxel Mapping . . . . . . . . . . . . . . . . . 10 3.3.2 Resulting Occupancy Grid . . . . . . . . . . . . . . . . . . . . . . 11 3.4 Points of Interest Detection . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.4.1 Clustering and Selection of Representative POIs . . . . . . . . . 12 Nearest Neighbors Analysis for Parameter Selection . . . . . . . 12 DBSCAN Clustering Algorithm . . . . . . . . . . . . . . . . . . 12 Medoid Selection for Representative POIs . . . . . . . . . . . . . 12 3.5 PathPlanning ................................. 15 3.5.1 Generating Paths Between Points of Interest . . . . . . . . . . . 15 Rapidly-exploring Random Tree (RRT) Algorithm . . . . . . . . 15 PathConcatenation.......................... 15 3.5.2 Path Smoothing with B-Spline Interpolation . . . . . . . . . . . 15 3.6 Camera Direction Optimization . . . . . . . . . . . . . . . . . . . . . . . 16 3.6.1 Optimal Direction Computation at Points of Interest . . . . . . 16 Candidate Direction Generation . . . . . . . . . . . . . . . . . . 16 Field of View (FOV) Determination . . . . . . . . . . . . . . . . 18 Optimal Direction Selection . . . . . . . . . . . . . . . . . . . . . 19 3.6.2 Direction Interpolation Between POIs . . . . . . . . . . . . . . . 19 3.7 Path Serialization for Rendering . . . . . . . . . . . . . . . . . . . . . . . 19 3.7.1 Camera Pose Representation . . . . . . . . . . . . . . . . . . . . 20 3.7.2 Rendering with Pre-trained NeRF Model . . . . . . . . . . . . . 20 3.7.3 VideoCompilation.......................... 20 2Chapter 1. Introduction 1. Integration of NeRF for Virtual Property Showcases: Develop a system that integrates NeRF technology with camera path planning to create a visually compelling and optimized property tour. 2. Path Planning and Camera Direction Optimization: Implement and optimize path planning algorithms, such as RRT, to generate smooth, efficient, and visually comprehensive camera paths. 3. Maximization of Scene Coverage: Ensure that the camera paths maximize the visibility of important areas of the property by optimizing both the spatial trajectory and the camera’s orientation. 4. Contribution to StageInHome’s Product Offerings: Deliver a solution that can be used as part of StageInHome’s AI-driven virtual staging products, enhancing the company’s ability to offer dynamic property tours to its clients. 3 Chapter 2 State of the Art This chapter provides an overview of the key concepts and related work that serve as the foundation for this thesis. The primary focus is on Neural Radiance Fields (NeRF), which is used for 3D scene representation, and various path planning algorithms. Furthermore, we explore the integration of NeRF with path planning techniques for virtual camera navigation and scene coverage optimization. The chapter is organized as follows: •Neural Radiance Fields (NeRF): An introduction to NeRF, its rendering process, and its advantages in 3D scene representation. •Path Planning: A comparison of classical and probabilistic path planning algorithms, emphasizing their relevance to virtual camera navigation. •Exploration in NeRF: Examination of existing work in applying NeRF for exploration in physical environments and the relevance of these techniques to virtual environments. 2.1 Neural Radiance Fields (NeRF) Neural Radiance Fields (NeRF) have revolutionized the field of computer vision and graphics by providing a novel approach to 3D scene representation and novel view synthesis. Introduced by [1], NeRF leverages deep neural networks to model complex 3D scenes with remarkable detail and realism. At its core, NeRF represents a scene as a continuous volumetric radiance field parameterized by a fully connected deep neural network. The network takes a 3D spatial coordinate (x,y,z)and a 2D viewing direction (ϕ,θ)as inputs and outputs the volume density σand the RGB color cat that point. Formally, the neural network can be described as: (σ,c) = NeRF(x,y,z,ϕ,θ;ω)(2.1) where: •(x,y,z)are the 3D spatial coordinates of the point in the scene. •(ϕ,θ)are the 2D angular coordinates representing the viewing direction. •σis the predicted volume density at the point. •cis the predicted RGB color. •ωdenotes the neural network’s parameters. 4Chapter 2. State of the Art NeRF employs a differentiable rendering process based on volume rendering techniques to synthesize novel views from arbitrary camera angles. For each pixel in the target view, a ray is cast into the scene, and points along the ray are sampled at discrete intervals. The color and density values predicted by the network for these sample points are accumulated using alpha compositing, a process that combines the colors and densities of the sampled points along a ray to compute the final pixel color, based on their opacity. Points closer to the camera block the light coming from further points, simulating realistic depth effects. The color along the ray is computed as: C(r) = N ∑ i=1 i−1 ∏ j=1 e−σjδj!(1−e−σiδi)ci(2.2) where: •C(r)is the final color of the ray r. •Ti=∏i−1 j=1e−σjδjrepresents the accumulated transmittance. •σiand ciare the density and color at the i-th sample point. •δiis the distance between consecutive sample points. NeRF is trained using a set of posed images of the target scene. The network parameters θare optimized to minimize the difference between the rendered pixel colors and the ground truth pixel colors across all training views. This optimization enables the network to learn an accurate and high-fidelity representation of the 3D scene. Some notable advantages of NeRF models are: •High-Quality Rendering: NeRF achieves state-of-the-art results in novel view synthesis, capturing fine geometric details and complex lighting effects. •Continuous Representation: Unlike discrete voxel grids or mesh-based models, NeRF provides a continuous and smooth representation of the scene, allowing for arbitrary resolution and detail. •Differentiable Rendering: The end-to-end differentiable rendering pipeline facilitates efficient training using gradient-based optimization methods. In the context of camera path planning and exploration, NeRF serves as an effective tool for modeling the 3D environment. By providing a detailed and continuous representation of the scene, NeRF enables the development of algorithms that can navigate and explore complex spaces with high precision. The ability to generate novel views allows for comprehensive coverage and visualization, making NeRF a pivotal component in advanced exploration systems. In addition to the original NeRF, several variations have been proposed to address its limitations. For example, mip-NeRF [2] improves efficiency by representing rays as cones rather than lines, reducing aliasing in the rendered images. Neural Sparse Voxel Fields (NSVF) [3] enhances memory and computational efficiency by using sparse voxel grids. Another promising alternative is Gaussian Splatting [4], which uses Gaussians to model the scene and provides more efficient training and rendering. 2.2. Path Planning 5 2.2 Path Planning Path planning is a fundamental aspect of autonomous navigation, enabling robots to determine feasible and efficient trajectories from a start point to a goal within an environment. Path planning algorithms can be broadly categorized into Classical Path Planning Algorithms and Probabilistic Path Planning Algorithms. Each category offers distinct methodologies and advantages suited to different application scenarios. 2.2.1 Classical Path Planning Algorithms Classical path planning algorithms, such as Dijkstra’s Algorithm [5] and A* [6], are well-established methods that operate on graph-based representations of the environment. These algorithms are deterministic and guarantee finding the shortest path in a weighted graph, provided certain conditions are met. Dijkstra’s Algorithm is a classic method for finding the shortest path between nodes in a graph. It systematically explores all possible paths from the start node, updating the shortest known distances to each node until the goal is reached [5]. While comprehensive, Dijkstra’s algorithm can be computationally intensive for large graphs due to its exhaustive search nature. The A* algorithm is an extension of Dijkstra’s algorithm that incorporates heuristics to improve search efficiency. By estimating the cost from the current node to the goal (using a heuristic function), A* prioritizes nodes that are more likely to lead to an optimal path, reducing the number of nodes explored compared to Dijkstra’s algorithm [6]. Classical algorithms excel in environments where the graph representation is welldefined and manageable in size. They provide optimal solutions with guaranteed performance metrics. However, their deterministic nature and reliance on predefined graph structures can limit their scalability and adaptability in dynamic or highdimensional spaces. 2.2.2 Probabilistic Path Planning Algorithms Probabilistic Path Planning Algorithms offer alternative approaches that handle uncertainty and complexity more effectively than classical methods. Prominent algorithms in this category include Rapidly-exploring Random Trees (RRT) [7] and Probabilistic Roadmaps (PRM) [8]. These algorithms are particularly suited for high-dimensional and complex environments where deterministic methods may falter. The RRT algorithm incrementally builds a space-filling tree by randomly sampling points in the search space and extending the tree towards these points [7]. This approach allows RRT to efficiently explore large, high-dimensional spaces and find feasible paths without exhaustive search. RRT is especially useful in robotics for navigating through cluttered or dynamic environments. Probabilistic Roadmaps (PRM) construct a roadmap by randomly sampling the configuration space and connecting nearby samples to form a graph [8]. Once the 6Chapter 2. State of the Art roadmap is built, it can be used to find paths between any two points by searching the graph. PRM is highly effective for multi-query scenarios where multiple pathfinding operations are required within the same environment. RRT and PRM excel in complex and high-dimensional spaces where classical algorithms may struggle. They offer greater flexibility and scalability, making them suitable for real-time applications and dynamic environments. However, these algorithms do not always guarantee the shortest path and may require fine-tuning of parameters to achieve optimal performance. While classical algorithms provide guaranteed optimal paths in well-defined graphs, probabilistic algorithms offer superior performance in complex and uncertain environments. The choice between these approaches depends on the specific requirements of the application, such as the need for optimality versus computational efficiency and adaptability. 2.3 Exploration and Path Planning in Neural Radiance Fields Recent advancements in Neural Radiance Fields (NeRF) have shown considerable potential in the realm of vision-based autonomous exploration and path planning. These systems capitalize on NeRF’s ability to reconstruct 3D environments using 2D images, enabling detailed scene understanding and navigation. Several papers have explored the integration of NeRF into robotic navigation, focusing on real-world exploration. Although these efforts center on physical robots, they provide valuable insights into NeRF-based exploration systems, which contrast with the virtual camera path planning approach employed in this project. In the work by Adamkiewicz et al., NeRF is used to facilitate vision-only robot navigation [9]. Their approach leverages NeRF’s ability to reconstruct and refine a scene dynamically as the robot navigates. This allows the robot to autonomously select views that maximize the information it gathers, improving scene reconstruction while simultaneously navigating. NeRF acts as both a map and a navigation tool, providing real-time feedback to the robot. Building upon the theme of robot navigation in unknown environments, Chen et al. also proposed CATNIPS, a method aimed at addressing collision avoidance through probabilistic scene representations derived from NeRF [10]. CATNIPS provides a framework for predicting collisions in unseen parts of the environment by using NeRF to represent the geometry of the scene probabilistically. This allows robots to navigate safely, even in partially observable or uncertain areas. Although CATNIPS focuses on real-world robotics, the insights into using NeRF for understanding scene geometry could inform future virtual exploration systems. Further exploration of NeRF’s utility in navigation is seen in Han et al., who introduce NVINS, a system that fuses NeRF with visual-inertial odometry (VIO) to improve robot navigation accuracy [11]. NVINS leverages NeRF data to provide more accurate pose estimation, significantly reducing drift and enhancing robustness in feature-sparse environments. The integration of NeRF with traditional navigation systems exemplifies the adaptability of NeRF in challenging real-world navigation tasks. In contrast to these works, which are designed for real-time robot navigation in uncertain and dynamically changing environments, this project focuses on the virtual exploration of a static NeRF model. The previous methods primarily deal with 2.4. Identified Gaps in the Research 7 physical aspects of navigation, such as collision avoidance, uncertainty quantification, and real-time scene reconstruction. These factors are critical in the context of robotics but are not relevant to the objectives of this project, which involves virtual camera path optimization for scene rendering. The gap in the literature lies in the development of NeRF-based systems tailored for virtual environments, where the emphasis is on pre-computed, optimized paths for showcasing and visualization, without concerns of physical movement or real-time adjustments. Thus, while existing research demonstrates NeRF’s potential in robotic exploration and path planning, there remains an open area for applying these techniques in non-physical, virtual contexts. This project addresses that gap, utilizing NeRF to predefine optimal camera paths in a fixed environment, specifically aimed at applications like property showcasing and virtual tours. 2.4 Identified Gaps in the Research While recent advancements in NeRF-based exploration have showcased significant potential in the context of robotic navigation, particularly for vision-only systems, there remains an unexplored area concerning virtual environments. The existing literature focuses predominantly on real-world applications where robots must autonomously navigate uncertain, dynamic environments. These systems emphasize critical factors like collision avoidance, pose estimation, and real-time navigation, which are essential in physical exploration. However, these studies do not fully address the needs of static, virtual environments where the objective is to optimize camera movement for scene visualization rather than physical navigation. In applications such as virtual property showcasing, the challenge shifts from avoiding obstacles and ensuring safe navigation to maximizing scene coverage, visual smoothness, and overall rendering quality. This project identifies a clear gap in the research: the absence of predefined, optimized camera paths designed specifically for virtual NeRF environments. The focus here is on maximizing the visual experience through careful camera path planning and orientation adjustment, without the constraints of real-time decision-making or physical movement. By adapting path planning algorithms to the unique demands of virtual navigation, this work aims to bridge that gap, offering a new perspective on how NeRF can be utilized for static scene exploration and visualization optimization. 9 Chapter 3 Methodology 3.1 High Level System Overview FIGURE 3.1: High level system block diagram This section presents the systematic approach developed to optimize camera paths and orientations for rendering 3D scenes using a Neural Radiance Field (NeRF) model. The methodology integrates several key components: data preparation, occupancy grid construction, points of interest detection, path planning, camera direction optimization, and path serialization for rendering. The primary objective is to generate a collision-free camera path that maximizes scene coverage, thereby enhancing the visual representation of significant areas within the 3D environment. An overview of the system architecture is illustrated in Figure 3.1 10 Chapter 3. Methodology 3.2 Data Preparation The initial step involves preparing the data extracted from a trained NeRF model. Two crucial outputs are obtained: •Point Cloud Export (PC): A point cloud (PC) is a collection of data points in space, representing the external surfaces of objects or scenes. The NeRF model is converted into a point cloud representation, typically in the .ply file format. A rendered view of an example scene can be visualized in Figure 3.2. This PC captures the 3D geometry of the scene, providing a detailed spatial framework for further processing. •Training Cameras Information (C): Cameras are represented as a set C, where each camera Cihas a translation vector Tidenoting its position. The positional and orientational data of the cameras used during the NeRF training process are extracted. This includes the locations and viewing directions, which are essential for understanding the initial perspectives from which the scene was captured. FIGURE 3.2: Single view of the exported point cloud (PC) for the kitchen scene. 3.3 Occupancy Grid Construction The occupancy grid serves as a discretized volumetric representation of the 3D environment, facilitating collision detection and efficient spatial queries. By overlaying a regular grid of voxels onto the Point Cloud (PC), the system distinguishes between occupied and free space, enabling precise navigation and visibility analysis. 3.3.1 Grid Overlay and Voxel Mapping Constructing the occupancy grid involves initializing the grid based on the spatial boundaries of the PC and mapping each point to its corresponding voxel within the grid. 3.4. Points of Interest Detection 11 •Grid Initialization: An occupancy grid Gis created by overlaying a 3D grid structure onto the PC. The grid resolution Ndetermines the number of divisions along each axis, balancing detail and computational efficiency. •Voxel Mapping: Each point p= (xp,yp,zp)in the PC is normalized relative to the bounding box of the scene and mapped to grid indices (i,j,k): i=xp−xmin xmax −xmin ×(N−1), (3.1) j=yp−ymin ymax −ymin ×(N−1), (3.2) k=zp−zmin zmax −zmin ×(N−1). (3.3) The voxel G(i,j,k)is then marked as occupied. 3.3.2 Resulting Occupancy Grid The mapping process results in a comprehensive occupancy grid that accurately represents the spatial distribution of objects within the environment. This grid differentiates between occupied and free space, facilitating various computational tasks such as collision detection and visibility analysis. FIGURE 3.3: Visualization of the constructed occupancy grid. The grid accurately maps the occupied regions based on the underlying Point Cloud data. 3.4 Points of Interest Detection Identifying Points of Interest (POIs) is crucial for ensuring that the camera path navigates through significant areas within the 3D scene. POIs represent key features and important regions that enhance the comprehensiveness and visual coverage of the rendered video. By clustering the extensive set of training camera positions, we can select representative POIs that effectively highlight the scene’s essential components. 18 Chapter 3. Methodology d=  cos(θ)cos(ϕ) sin(θ)cos(ϕ) sin(ϕ) (3.5) where θ∈[0,2π)and ϕis varied to introduce upward and downward tilts. Field of View (FOV) Determination To ascertain whether a scene point plies within the camera’s FOV, the algorithm defines the camera’s frustum—a pyramidal volume characterized by the horizontal and vertical FOV angles. Frustum Plane Calculation The frustum is bounded by four planes: left, right, top, and bottom. Each plane is defined by a normal vector calculated based on the camera’s orientation and FOV angles. For a given camera direction c, the normals nl,nr,nt,nbfor the left, right, top, and bottom planes are computed as follows: nl=rotate(c,u,FOVh 2)(3.6) nr=rotate(c,u,−FOVh 2)(3.7) nt=rotate(c,r,FOVv 2)(3.8) nb=rotate(c,r,−FOVv 2)(3.9) where: •uand rare the up and right vectors relative to the camera’s orientation. • rotate(v,a,α)denotes the rotation of vector varound axis aby angle αusing Rodrigues’ rotation formula. FIGURE 3.8: Illustration of the camera frustum planes. The frustum defines the camera’s field of view based on horizontal and vertical FOV angles. Rodrigues’ rotation formula is given by: rotate(v,a,α) = vcos(α) + (a×v)sin(α) + a(a·v)(1−cos(α)) (3.10) 3.7. Path Serialization for Rendering 19 Point Inclusion Test A point pis considered within the FOV if it satisfies the following conditions relative to all frustum planes: nl·(p−c)≥0 (3.11) nr·(p−c)≥0 (3.12) nt·(p−c)≥0 (3.13) nb·(p−c)≥0 (3.14) where cis the camera position. If all inequalities hold, plies within the FOV. Optimal Direction Selection To identify the optimal camera direction at each POI, an optimization technique is employed that evaluates scene coverage across all candidate directions. The direction d∗that maximizes the number of points within the FOV is selected as the optimal orientation: d∗=argmax di ∑ j I(pj∈FOV(di)) (3.15) where Iis the indicator function, which equals 1 if pjis within the FOV and 0 otherwise. Optimization Technique: The selection of d∗involves evaluating each candidate direction’s coverage by counting the number of scene points that fall within the defined FOV. This process ensures that the camera is oriented in a direction that maximizes visibility and comprehensiveness of the scene. By systematically assessing all possible directions, the algorithm guarantees that the chosen orientation provides the most informative view from each POI. 3.6.2 Direction Interpolation Between POIs To ensure smooth transitions between POIs, camera orientations for intermediate waypoints are interpolated using Bézier curves. This approach creates a visually appealing and fluid camera trajectory, enhancing the overall quality of the rendered video. Bézier Curve Interpolation is utilized to smoothly transition camera orientations between POIs. By defining control points based on the optimized directions at each POI, Bézier curves generate intermediate orientations that maintain continuity and natural movement. This method is favored due to its popularity in video editing software, where it is commonly used to achieve smooth and controllable transitions. 3.7 Path Serialization for Rendering With the camera path fully planned and optimized, comprising a series of camera poses defined by Camera_to_world matrices, the next step is to render the visual output. Each camera pose in the path specifies the spatial position and orientation of the camera within the 3D environment, forming the foundation for generating the final visualization video. 20 Chapter 3. Methodology 3.7.1 Camera Pose Representation Each waypoint along the camera path is characterized by a Camera_to_world matrix, which encapsulates both the translation and orientation of the camera in 3D space: •Camera_to_world Matrix Ci: A 4x4 transformation matrix that defines the camera’s position and orientation. It consists of a rotation matrix and a translation vector: Ci=RiTi 0⊤1(3.16) where: – Riis a 3x3 rotation matrix representing the camera’s orientation. – Ti= (xi,yi,zi)⊤is the translation vector specifying the camera’s position in world coordinates. These matrices collectively define the camera’s viewpoint at each waypoint, ensuring precise spatial placement and optimal orientation for comprehensive scene coverage. 3.7.2 Rendering with Pre-trained NeRF Model To generate the final visualization video, the pre-trained Neural Radiance Field (NeRF) model is utilized to render frames corresponding to each camera pose. The rendering process involves the following steps: 1. Input Preparation: For each camera pose Ci, the corresponding Camera_to_world matrix is formatted into a structure compatible with the NeRF rendering pipeline. This typically involves defining the camera’s position and orientation parameters required by the NeRF software. 2. Frame Rendering: The formatted camera pose Ciis fed into the NeRF model, which synthesizes a novel view from the specified viewpoint. This results in the generation of a rendered frame (image) that captures the scene from the camera’s position and orientation. 3. Frame Collection: Each rendered frame is saved sequentially, corresponding to the order of camera poses along the planned path. 3.7.3 Video Compilation After rendering frames for all camera poses along the path, these images are sequentially joined to create the final video. This compilation process involves: •Frame Sequencing: Arranging the rendered frames in the order of their corresponding camera poses to ensure smooth temporal progression. •Video Encoding: Converting the sequence of images into a video format using standard video encoding tools, resulting in a coherent and continuous visualization of the camera’s traversal through the 3D environment. 21 Chapter 4 Results This section presents the outcomes of the implemented system, comparing a baseline Rapidly-exploring Random Tree (RRT) path with the optimized path that incorporates smoothing and camera direction optimization. Before diving into the detailed results and performance metrics, we will first provide an overview of the dataset and data preparation process, followed by a summary of the system’s implementation. These foundational details are essential for understanding the context in which the results were produced. Afterward, key metrics will be presented to evaluate and compare the effectiveness of both the RRT and optimized paths. 4.1 Dataset and Data Preparation The system was developed and tested using the publicly available Kitchen Dataset, provided by Nerfstudio. Which provides a comprehensive collection of images and associated camera information necessary for training the NeRF model. The dataset comprises over 150 high-resolution images capturing various perspectives of a kitchen environment, facilitating accurate scene reconstruction. Utilizing the trained NeRF model, a point cloud containing over one million points was extracted, representing the detailed geometry of the kitchen. This dense point cloud serves as the foundation for building the occupancy grid and performing subsequent camera path optimization. 4.2 Implementation The camera path optimization system was developed using Python, leveraging key libraries such as NumPy for numerical computations, SciPy for scientific computing, Open3D for 3D data processing, and Scikit-learn for machine learning algorithms. These tools facilitated efficient handling of point clouds, occupancy grids, and the implementation of optimization and clustering algorithms essential for the system’s functionality. The architecture is modular, comprising classes responsible for managing occupancy grids, optimizing camera directions, planning paths, and serializing data for rendering. For a comprehensive overview of the system’s implementation, including class structures and specific algorithms, please refer to Appendix A. 22 Chapter 4. Results To assess the effectiveness of the developed system, several metrics are introduced that provide a quantitative analysis of the camera path’s quality and the overall scene coverage. These metrics offer insights into the smoothness of camera movements and the visibility of key areas within the scene, ensuring that the virtual tour is both visually appealing and comprehensive. By comparing the baseline RRT path with the optimized path, we can objectively measure improvements in path planning and scene visualization. The following sections outline these metrics in detail and present the corresponding results. 4.3 Path Smoothness The smoothness of the camera directions along the path is a crucial metric for evaluating the quality of a planned path. A smooth path ensures that there are minimal abrupt changes in the camera’s orientation, leading to a more visually pleasant experience. To quantify this, we introduce the Camera Direction Smoothness Cost, which measures the angular changes between consecutive camera directions along the path, normalized by the total number of waypoints. Let dibe the direction vector of the camera at waypoint i, for i=1, 2, . . . , N where Nis the total number of valid waypoints. The smoothness cost is computed as the sum of the angular changes between consecutive direction vectors along the path. The angle θibetween two consecutive direction vectors diand di−1is calculated using the dot product formula: θi=cos−1di·di−1 ∥di∥∥di−1∥(4.1) where ·denotes the dot product, and ∥d∥represents the magnitude of the vector. The total smoothness cost is the sum of all angular changes divided by the number of valid waypoints N: Smoothness Cost =1 N−1 N ∑ i=2 θi(4.2) This normalization ensures that the smoothness cost is independent of the path length. 4.3.1 Comparative Analysis: RRT Path vs. Optimized Path To assess the impact of path smoothing and direction optimization, we compare the normalized smoothness cost of the initial RRT path (unsmoothed) and the final optimized path. The results show a significant reduction in the smoothness cost, demonstrating the effectiveness of the optimization process in creating a more fluid camera movement. Method Smoothness Cost RRT Path 0.7592 Optimized Path 0.0592 TABLE 4.1: Camera Direction Smoothness Cost Comparison The unsmoothed RRT path exhibits a higher smoothness cost, indicating frequent and abrupt changes in camera orientation between consecutive waypoints. 4.4. Scene Coverage Metric 23 This is expected due to the RRT algorithm’s nature, which does not inherently prioritize smooth transitions between sampled waypoints. In contrast, the optimized path, which incorporates B-spline smoothing and camera direction optimization, achieves a significantly lower smoothness cost. The optimization reduces sharp directional changes, resulting in a smoother and more visually appealing camera movement through the 3D environment. This reflects the importance of incorporating both positional and directional smoothing techniques in creating high-quality camera paths for applications like virtual tours or scene rendering. 4.4 Scene Coverage Metric Scene coverage is a critical metric in path planning, particularly when the objective is to maximize the visibility of important areas within a 3D environment. In our experiment, we define the scene coverage metric as the number of visible voxels (occupied cells) in the environment that are observed by the camera along the planned path. The goal is to ensure that the path allows the camera to view as much of the scene as possible. We compute this metric using an occupancy grid of the environment, which discretizes the 3D space into a grid of voxels. The camera’s field of view (FOV) is calculated at each Point of Interest (POI) along the path, and the number of visible occupied cells from that point is recorded. The total scene coverage is then derived by summing the visible cells across all POIs. 4.4.1 Scene Coverage Computation Given the complexity of determining how much of the scene is visible from various points, we evaluate the scene coverage for each path by calculating the number of visible occupied cells across the grid. For each POI, the camera’s FOV is used to check which cells fall within the FOV’s boundaries using the following process: • For each waypoint along the path classified as a POI, we extract the camera’s position, FOV, aspect ratio, and direction vector. • We use a field-of-view checker to determine which voxels in the occupancy grid are visible from each POI. • The total number of visible cells is calculated by accumulating all visible cells across the POIs. We compute the scene coverage in two ways to ensure comprehensive evaluation: 1. Raw Scene Coverage: The total number of visible occupied cells without any normalization. 2. Scene Coverage Normalized by Occupied Cells: The number of visible occupied cells divided by the total number of occupied cells in the environment. 24 Chapter 4. Results Method Raw Scene Coverage Normalized by Occupied Cells RRT Path 211,193 1.5447 Optimized Path 253,300 1.8526 TABLE 4.2: Comparison of Scene Coverage for RRT and Optimized Paths 4.4.2 Results and Discussion Table 4.2 compares the scene coverage for the RRT-generated path and the optimized path. As shown in Table 4.2, the optimized path achieves a higher scene coverage in both metrics compared to the RRT path. Specifically: • The Raw Scene Coverage for the optimized path is 253,300 visible cells, compared to 211,193 for the RRT path. • When Normalized by Occupied Cells, the optimized path covers 1.8526 times the occupied cells in the environment, while the RRT path covers 1.5447 times. 4.4.3 Challenges in Scene Coverage Evaluation While the optimized path demonstrates better scene coverage, it is important to note that finding an objective, meaningful metric for scene coverage is challenging. The slight differences in normalized values between the RRT and optimized paths may not fully capture the subjective experience of better coverage, especially when visual quality and camera smoothness play crucial roles in evaluating the final output. These close values indicate that both paths are effective in covering the environment, and the choice of metrics significantly influences how the coverage is perceived. For example, normalizing by occupied cells and grid resolution provides different perspectives on the same data. Therefore, although the optimized path performs better, both paths achieve high levels of scene visibility, indicating that RRT is still a viable option depending on the application requirements. 4.5 Coverage Distribution To better understand how the camera path covers the environment, we compute a Scene Coverage Heatmap. This heatmap provides a visual representation of the distribution of observed areas in the 3D scene, allowing us to identify which regions of the environment are viewed more frequently by the camera and which areas are left unobserved. The heatmap is constructed by counting the number of times each voxel in the occupancy grid is visible from the camera’s field of view (FOV) at various Points of Interest (POIs) along the path. The higher the value in a voxel, the more times that part of the environment has been viewed by the camera. 4.5.1 Heatmap Computation The scene coverage heatmap is generated by the following process: • For each POI in the camera path, the camera’s position, FOV, aspect ratio, and direction vector are extracted. 4.5. Coverage Distribution 25 • The visibility of each voxel in the occupancy grid is checked using a field-ofview (FOV) checker, which determines whether the voxel is within the camera’s FOV from that POI. • The heatmap is incremented for each voxel that is visible from a POI. The heatmap values reflect how many times a voxel is observed by the camera. This process is repeated for every POI along the path, resulting in a comprehensive heatmap that shows how the camera explores the 3D environment over time. Areas with higher values in the heatmap indicate regions that are viewed more frequently by the camera, while areas with lower values represent less frequently viewed or unobserved regions. 4.5.2 Results and Visualization The following figures show the coverage heatmap for the RRT-generated path and the optimized path. These heatmaps allow for direct comparison of how well each path covers the scene. FIGURE 4.1: Scene Coverage Heatmap for the RRT Path. As shown in Figures 4.1 and 4.2, the optimized path results in a more evenly distributed coverage compared to the RRT path. The optimized path shows fewer areas with extremely high values, indicating a smoother and more consistent coverage of the scene, whereas the RRT path has regions that are either viewed excessively or barely covered. 26 Chapter 4. Results FIGURE 4.2: Scene Coverage Heatmap for the Optimized Path. 4.5.3 Discussion The heatmap visualization offers a powerful way to analyze the effectiveness of each path in covering the scene. The RRT-generated path tends to focus on certain regions, potentially over-covering them while leaving other regions relatively unexplored. In contrast, the optimized path provides a more uniform coverage, ensuring that the camera observes all important areas of the environment with fewer redundant views. This metric complements the previous scene coverage evaluation by showing not only how much of the scene is covered, but also how well the coverage is distributed across the environment. The results suggest that the optimized path is more effective in providing a balanced exploration of the scene, avoiding both excessive focus on certain areas and missing important regions entirely. 4.6 Cost Function Analysis To optimize camera directions at each Point of Interest (POI), we define a Scene Coverage Cost Function. This cost function evaluates how well a particular camera direction maximizes the number of visible occupied cells within the camera’s field of view (FOV) at each POI. By exploring different directions, the cost function allows us to find the optimal orientation that provides the highest scene coverage. Figure 4.3 shows the cost function evaluated for various candidate camera directions at a specific POI. The cost function displays a clear peak, corresponding to the direction that maximizes scene coverage, and a trough, indicating the direction that results in minimal coverage. This visualization demonstrates the effectiveness of the cost function in guiding the optimization process to find the best camera orientations. 4.6. Cost Function Analysis 27 FIGURE 4.3: Scene Coverage Cost Function for a specific POI, showing a clear maximum and minimum. 4.6.1 Best and Worst Views To further illustrate the meaning of the cost function, we rendered the camera views corresponding to the directions with the highest and lowest cost values. These renderings provide a visual comparison of the best and worst views, highlighting the impact of camera orientation on scene coverage. Best View Figure 4.4 shows the view corresponding to the direction with the highest scene coverage. This direction maximizes the number of visible occupied cells, resulting in a comprehensive and optimal view of the scene. The best view demonstrates how the cost function can successfully guide the camera to a direction that provides maximum visibility of the environment. Worst View Figure 4.5 shows the view corresponding to the direction with the lowest scene coverage. This direction results in minimal visibility, with only a small portion of the scene visible to the camera. The worst view demonstrates the importance of optimizing camera orientation to avoid such poor coverage scenarios. The cost function analysis confirms that the optimization process is effective in selecting the best camera orientation for maximizing scene coverage. The clear peak in the cost function corresponds to a highly visible and informative view of the environment, as demonstrated by the best view rendering. Conversely, the trough in the cost function reflects a poor orientation with minimal coverage, as shown in the worst view. These visualizations reinforce the utility of the cost function in optimizing camera directions for comprehensive scene exploration. 34 Appendix A. Implementation Details A.2.10 POIExtractor Purpose: Extracts Points of Interest (POIs) from training camera data using clustering algorithms. Key Methods: •compute_pois: Identifies and selects representative POIs from the dataset. A.2.11 PointCloudConverter Purpose: Converts point cloud data into an occupancy grid and exports it for visualization. Key Methods: •load_point_cloud: Loads point cloud data from a file. •convert_to_occupancy_grid: Transforms the point cloud into an occupancy grid. •save_occupancy_grid_as_obj: Exports the occupancy grid as an .obj file for visualization purposes. A.3 Noteworthy Implementation Details A.3.1 Parallel Processing with Multiprocessing The FOVBatchProcessor class employs Python’s multiprocessing library to perform FOV checks across multiple directions concurrently. By leveraging multiple CPU cores, the system significantly reduces computation time, enabling real-time or near-real-time processing even with large datasets. A.3.2 Comprehensive Camera Orientation Optimization The CameraDirectionOptimizer class meticulously evaluates multiple candidate directions at each POI to identify the optimal orientation that maximizes scene coverage. By combining geometric computations of the camera frustum with optimization techniques, the system ensures that camera movements are both comprehensive and natural. A.3.3 Serialization for NeRF Rendering The PathSerializer class translates the optimized camera path into a JSON format compatible with NeRF rendering pipelines. This serialization process is streamlined to facilitate seamless integration with rendering tools, ensuring that the final visualization accurately reflects the planned camera movements. A.4 Development Environment The system was developed and tested on a machine running Ubuntu 20.04 LTS, equipped with an NVIDIA RTX 3080 GPU to accelerate NeRF rendering processes. The combination of Python for high-level logic and efficient libraries ensures both flexibility and performance in the system’s execution. 35 Bibliography [1] Ben Mildenhall et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”. In: European Conference on Computer Vision (ECCV). 2020. [2] Jonathan T. Barron et al. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. 2021. arXiv: 2103.13415 [cs.CV].URL:https://arxiv. org/abs/2103.13415. [3] Lingjie Liu et al. Neural Sparse Voxel Fields. 2021. arXiv: 2007.11571 [cs.CV]. URL:https://arxiv.org/abs/2007.11571. [4] Bernhard Kerbl et al. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. 2023. arXiv: 2308.04079 [cs.GR].URL:https://arxiv.org/abs/2308. 04079. [5] Edsger W Dijkstra. “A note on two problems in connexion with graphs”. In: Numerische Mathematik 1.1 (1959), pp. 269–271. [6] Peter E Hart, Nils J Nilsson, and Bertram Raphael. “A formal basis for the heuristic determination of minimum cost paths”. In: IEEE Transactions on Systems Science and Cybernetics 4.2 (1968), pp. 100–107. [7] Steven M. LaValle. “Rapidly-exploring random trees : a new tool for path planning”. In: The annual research report (1998). URL:https://api.semanticscholar. org/CorpusID:14744621. [8] L.E. Kavraki et al. “Probabilistic roadmaps for path planning in high-dimensional configuration spaces”. In: IEEE Transactions on Robotics and Automation 12.4 (1996), pp. 566–580. DOI:10.1109/70.508439. [9] Michal Adamkiewicz et al. Vision-Only Robot Navigation in a Neural Radiance World. 2022. arXiv: 2110.00168 [cs.RO].URL:https://arxiv.org/abs/2110. 00168. [10] Timothy Chen, Preston Culbertson, and Mac Schwager. “CATNIPS: Collision Avoidance Through Neural Implicit Probabilistic Scenes”. In: IEEE Transactions on Robotics 40 (2024), 2712–2728. ISSN: 1941-0468. DOI:10 . 1109 / tro . 2024.3386394.URL:http://dx.doi.org/10.1109/TRO.2024.3386394. [11] Juyeop Han et al. NVINS: Robust Visual Inertial Navigation Fused with NeRFaugmented Camera Pose Regressor and Uncertainty Quantification. 2024. arXiv: 2404.01400 [cs.RO].URL:https://arxiv.org/abs/2404.01400.