Repositorio Institucional de Documentos

A machine learning approach for transient imaging reconstruction Miguel Ángel Cosculluela Gracia Trabajo de fin de Máster en Modelización e Investigación Matemática, Estadística y Computación Universidad de Zaragoza Directores del trabajo: Dr. Julio Marco Murria y Prof. Dr. Diego Gutiérrez Pérez 9 de febrero de 2021 Abstract The recent advances in non-line-of-sight imaging have made it possible to reconstruct scenes hidden around a corner, with potential applications in e.g. autonomous driving or medical imaging. By operating at frame rates comparable to the speed of light, recent virtual-wave propagation methods leverage the temporal footprint of indirect light transport at a visible auxiliary surface to take virtual photos of objects hidden from the observer. Despite these advances, these methods have a critical computational bottleneck: The reconstruction quality and the computational performance are highly dependent on the resolution of the capture grid, which is typically discretized in space and time, leading to high processing and memory constraints. Inspired by recent machine learning techniques, in this work we propose a new computational imaging method to address these limitations. For this purpose we propose to learn implicit representations of the captured data using neural networks, allowing us to convert the discrete space of the captured data into a continuous one. However, working directly with the captured data is a complex task due to its huge size and its high dynamic range values. In order to avoid these problems, we leverage recent wave-based phasor-field imaging methods to transform the time-resolved captured data into sets of 2D complex-valued fields (i.e. phasor fields) at different frequencies, which provides a more favorable representation for machine learning methods. Under our implicit representation formulation, we analyze the performance of different neural network models to represent the complex structure of phasor fields, starting from simpler representations, and iteratively providing more powerful models to add support for the complexity of the data. We demonstrate how recent machine learning techniques based on multilayer perceptrons with sine activation functions are capable of representing phasor fields analytically in both spatial and temporal frequency domains, and integrate them into the phasorfield framework to reconstruct hidden geometry. We finally test this neural model in different scenes, and measure its performance at higher resolutions not seen by the captured data. We show how the model is able to analytically upsample all dimensions, and demonstrate how our implicit representation additionally works as a denoiser of the source discretized phasor field. iii iv Abstract Resumen Los recientes avances en imagen non-line-of-sight han hecho posible la reconstrucción de escenas ocultas a través de una esquina, con la potencial aplicación en conducción autónoma o imagen medica. Al operar con fotogramas por segundo cercanos a la velocidad de la luz, recientes métodos de propagación virtual de ondas aprovechan la huella temporal del transporte de luz en una superficie auxiliar para tomar fotones virtuales de objetos ocultos al observador. A pesar de estos avances, estos métodos tienen un cuello de botella critico: La calidad reconstrucción y el coste de computo son altamente dependientes de la resolución de la malla de captura, la cual suele esta discretizada en espacio y tiempo, lo que conlleva grandes limitaciones de procesamiento y memoria. Inspirados en las recientes técnicas de aprendizaje automático, en este trabajo proponemos un nuevo método de imagen computacional para hacer frente a estas limitaciones. Para este propósito proponemos aprender representaciones implícitas de los datos capturados usando redes neuronales, permitiéndonos convertir el espacio discreto de los datos capturados en un espacio continuo. Sin embargo, trabajar directamente con los datos capturados es una tarea compleja debido a su gran tamaño y al alto rango dinámico de sus valores. Para evitar estos problemas aprovechamos el reciente método de imagen de los campos de fasores basados en ondas para transformar los datos capturados resueltos en tiempo en un conjunto de campos 2D de valores complejos (como son los campos de fasores) a diferentes frecuencias, lo cual provee una representación más favorable para los métodos de aprendizaje automático. Siguiendo nuestra formulación de representación implícita, hemos analizado el rendimiento de diferentes modelos de redes neuronales para representar la compleja estructura de los campos de fasores, empezando por representaciones simples, y proporcionando de forma iterativa modelos más potentes para añadir soporte para la complejidad de los datos. Demostramos como las técnicas recientes de aprendizaje automático basadas en preceptrones multicapa con funciones de activación sinusoidales son capaces de representar un campo de fasores analíticamente en los dominios espaciales y temporales, e integrarlas dentro del marco de los campos de fasores para reconstruir geometría oculta. Finalmente probamos este modelo de red neuronal con diferentes escenas y medimos su desempeño con mayores resoluciones que no han sido usadas en el entrenamiento. Mostramos como el modelo es capaz de generar más muestras en todas las dimensiones y demostramos como nuestra representación implícita además funciona como un método para eliminar ruido del campo de fasores discretizado. v vi Resumen To my family and friends in the laboratory that helped me during the realization of this master’s thesis. I would like to thank my directors, who orientated me, and my partner who supported me during this time and who has given me the strength to keep on. vii Contents Abstract iii Resumen v 1 Introduction 1 1.1 ThesisBackground ................................ 3 2 Related Work 5 2.1 Non-line-of-sight (NLOS) transient imaging . . . . . . . . . . . . . . . . . . . 5 2.2 Transientrendering................................ 6 2.3 Upsamplingmethods ............................... 6 3 Background 9 3.1 Neural network optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1.1 Multilayer perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.1.2 Activation functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.3 Gradient descent optimization . . . . . . . . . . . . . . . . . . . . . . 11 3.1.4 Stochastic gradient descent . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2 Non-line-of-sight (NLOS) transient imaging . . . . . . . . . . . . . . . . . . . 12 3.2.1 Phasorfields ............................... 13 4 Implicit representation of phasor fields 15 4.1 Neural implicit representation of phasor fields . . . . . . . . . . . . . . . . . . 17 4.2 Datatransformation................................ 21 5 Results 25 6 Conclusions and future work 31 ix 6Chapter 2. Related Work 2.2 Transient rendering In NLOS one of the most valuable tools that help to test and develop new methods is the simulation of the capture system and NLOS scenes and, in our case, the possibility to generate data for machine learning algorithms. The simulation is done with a rendering process. Concretely, the rendering process simulates the light and its interactions with the matter in a virtual scene and capture the light with virtual sensors. The standard rendering process used in most of the applications such as cinema or video games generates images in 2D with the light in the scene integrated. This type of render is known as steady-state rendering. Other type is the transient rendering methods which allow simulating how the light moves through a scene, rendering videos at frame rates comparable to the speed of light. However, adding the temporal dimension in transient rendering is not always possible. This is because some applications and methods developed for steady-state rendering cannot be applied easily or, in some cases, at all. To implement a transient renderer some works presented new time-resolved light transport equations [24, 25]. Other works generalize Monte-Carlo methods for steady-state rendering as path tracing to a time-resolved version [26, 13]. Other algorithms as photon mapping were also generalized to time-resolved versions [27]. The use and research in transient rendering have grown with the development of NLOS transient imaging field. Its use is extremely useful in the NLOS field due to it helps to develop and test NLOS methods. This is due to the possibility of generating transient profile in controlled conditions without the restrictions of real capture hardware and use them as reference data. Following this last idea it is possible to generate datasets and ground truth for deep learning methods. For example the work by Marco et al. [28] uses a transient renderer to generate a time-of-flight (TOF) dataset and train a deep learning model to reduce the error in TOF cameras. Other interesting work by Liang et al. [29] presented a compression method of transient profiles with a deep learning model trained with synthetic transient data. Also, the work by Galindo et al. [30] presented a public dataset with NLOS scenes. In contrast, some NLOS methods do not use the complete light paths but only the first bounce [31, 32, 33]. This type of rendering is faster and the work by Chopite et al. [34] uses this type of rendering to generate a bigger dataset and train a deep learning model to reconstruct NLOS scenes from their transient profiles. 2.3 Upsampling methods A capture process by definition cannot obtain the real shape of the space but only get a certain number of samples (which is known as the resolution). Having a higher resolution gives more accuracy of the measured space. However, the resolution is typically limited by the capture hardware or physical restrictions in the capture process. When the resolution is large enough, the original shape can be obtained with a simple linear interpolation because the changes between samples practically follow a straight line. But in most cases the resolution is too low and obtaining the original shape is a hard problem. With the evolution of neural networks several works have demonstrated their capabilities of understanding the structure of the data and being able to upsample them. For this propose, there are two main approximations. The first approach consists in upsampling to a higher resolution for a fixed size or scale rate (e.g. duplicating resolution). The second one uses the neural network to, for a concrete input like spatial coordinates, time or frequency, get the value of the original data or, if the input is previously unknown, hallucinate new samples. A machine learning approach for transient imaging reconstruction 7 The first approach is commonly used in image super-resolution. With the idea of upsampling for fixed sizes, several neural networks models have been developed from models which only need one low resolution image as an input [8] to models which leverage the information from multiple images such as video frames [35]. If the upsampling is done over the temporal domain instead of the spatial one the effect is super-slow motion videos [36]. The second approach is more flexible than the first one because it is possible to sample in concrete zones or at any desired resolution. However, they have the cons that their models are not general and, for each scene, a new training is required. Training and sampling a model have two main ways to be implemented. In the first way the model has as an input the scene or data (or a transformation of them) and the coordinates to be sampled. This case have been tested for compression models of textures images, obtaining a latent space of them. Then the authors can decompress the latent space by sampling with neural networks [37, 9] for coordinates in the original textures or for new ones. The other way fits the scene directly in the neural network. This is known as implicit representation and has the advantage that, once the neural network is trained, the training data is no more used. This type of networks has as an input the different types of coordinates (spatial, temporal, angular, etc.) and, as the output, the corresponding values for that coordinates. Like the previous methods, this type of models can be used to generate samples in previous unknown positions. A useful case is learning 3D scenes [38] which, in render applications, avoids the requirement of rendering new views. Closer to our approach there are recent similar works which simplify the training process and improve the results [10, 11] presenting a new method to transform the coordinates and improving significantly the results. Furthermore, the work of Sitzmann et al. [12] shows the possibilities of using periodical functions instead of the classical activation one for learning high frequency details in several implicit representation problems such as image or video representation. 8Chapter 2. Related Work Chapter 3 Background This chapter describes the main mathematical, physical, and computational aspects that this work builds upon. Firstly we introduce a basic knowledge on neural network explaining them and the model used in this work with the most common optimization method used with neural networks. Secondly we introduce the non-line-of-sight problem and explain the theory of phasor fields, a state-of-the-art method for non-line-of-sight reconstruction. 3.1 Neural network optimization Neural networks are learning systems inspired in human neurons and their interconnections. They are able to learn a transformation and understand the relationship between the input and output data used in the training process. The capacity of learning the transformation lies on the non-linearities (small changes in the input lead to big changes in the output) that networks are able to find and learn from a dataset. One of the main capabilities of the neural networks is the generalization, since once they are trained they can be used with new unknown data with a really low computational cost. An example of this are the classification networks that can be used even in low powerful smartphones with their camera. However, this powerful method has a strong requirement for the majority of applications as neural networks require datasets with hundreds of thousands of examples. A neural network is formed by a set of functions called neurons which are interconnected with each other. Each neuron has multiple inputs with which operates, propagating the result to the next neurons. This process is repeated from the first neurons that compute the input to the last ones which generate the output. This sequential execution allows ordering the neurons in layers. There are basically three types of layers: the input layer, the output layer and the hidden ones which are all the layers in between the input and the output. The number of these hidden layers is variable and can be finetuned depending on the problem which the neural network is trying to solve. The same strategy can be followed regarding the number of neurons. Both are hyper-parameters that need to be obtained experimentally. In the last years, the development of neural networks has derived in a specialization of architectures and different types of networks have shown better achievements for certain types of data or problems. For example, some architectures have shown high capacity generating a compressed version of the input and also decompressing it to the original size [29]. Some works have tested this capability, adding the possibility to obtain values of the original data 9 10 Chapter 3. Background for certain coordinates [9]. Other architectures have shown their capability learning implicit representations of a desired scene or geometry and being able, once trained, to recover the full original data by evaluating the neural network in the points of the original data [10]. Moreover, in these examples, the neural networks have shown the ability to hallucinate new unknown points, recovering more resolution than the original. In this work both examples can be understood as a solution to our problem but we will focus on the second approach and learn an implicit representation of the data using a multilayer perceptron architecture. 3.1.1 Multilayer perceptron A perceptron [39] is a unique neuron defined as: y=f n ∑ i=1 xiwi+b!,(3.1) where fis a non-linear activation function, like sigmoid or a hyperbolic tangent function, ware the weights of the perceptron, xare the inputs and bis the bias. This equation can be expressed in a matricial form: y=fxwT+b.(3.2) Neurons can be organized forming layers. The number of neurons in each layer is decided depending on the number of the outputs of the layer since each neuron has only one output value. If multiple layers are connected like in the figure 3.1, the architecture is called multilayer perceptron. The number of layers and the number of neurons in each one are obtained experimentally as hyper-parameters except for the last layer, whose number of neurons is equal to the number of outputs values. Figure 3.1: Multilayer perceptron scheme. Image from [40]. A machine learning approach for transient imaging reconstruction 11 3.1.2 Activation functions The capacity of the neural networks to solve difficult problems lies on the non-linearites that they are able to learn. They can learn them due to the non-linearites of its activation functions. The activation function basically transform the output of the neuron depending on certain conditions, which can change the whole distribution of the intermediate data. There are multiple activation functions such as hyperbolic tangent (equation 3.3), rectified linear unit (ReLU) [41] (equation 3.4), sigmoid (equation 3.5). tanh(x) = ex−e−x ex+e−x(3.3) ReLU(x) = 0 if x≤0 xif x>0(3.4) sigmoid(x) = 1 1+e−x(3.5) The type of function to use depends on the problem and the characteristics of the problem to solve. The selection of the activation function could be understood as an hyper-parameterization.. Despite the necessity of testing different activation functions, the number of candidates can be reduced based on the type of function that has been used for similar problems in previous works. For example, one of the most used activation functions to work with images is the ReLU function since it only propagates positives values. This is specially interestring when the output is an image because the standard range to work with them is in [0,1]. The characteristics of the activation functions help the networks to focus on the resolution of the problem. In contrast, selecting a bad activation function can diminish and hinder the convergence of the training process. 3.1.3 Gradient descent optimization The optimization process is done through the minimization of a function. In deep learning the function to minimize is called the loss function and measures the performance of the network over a task. The perfect solution would be to find the global minimum of the loss. However, that could be impossible due to the complexity of the parameter space. On the other hand, finding a local minimum is an easier task that can be done with gradient descent optimization methods. As a simplification, we can denote the loss function as y=f(x)which derivative is f0(x) = dy/dx. It is known that the derivative gives the slope of fat the point x. With the direction of the slope we know in which direction the function will be minimized and if the derivative is equal to 0 then the function is in a minimum. fcan be defined with several parameters in the form of x={x1,x2,...,xn}. The derivative of fmust be done with partial derivatives ∂f(x)/∂xi. The gradient of fdenoted as ∇f(x)will be the vector with all the partial derivatives. Like the gradient gives the slope of every parameter we can update them in the negative direction of the slope and descending iteratively in the gradient. The updating function can be denoted as: xt+1=xt−ε∇f(xt),(3.6) where xtis xvalues in the actual iteration and εis a small value called learning rate that controls the speed of the descent. The learning rate can not be too high because when the function is near to the minimum a huge learn rate could prevent the function to find the local 12 Chapter 3. Background minimum. On the other hand if the learning rate is too small the needed steps for getting the minimum point will be so high that the learning process would take too much time. 3.1.4 Stochastic gradient descent The previous definition hold for a loss with a single training data. However, in machine learning is typical to have a hundred of thousands of training data. In that case, the loss function can be understood as a sum over the loss of all training data. Considering the parameters of the model as θand any loss functions for a single data as Lwe can denote the loss Lfor all the example as L(θ) = E[L(x,y,θ)] = ∑n i=1Lx(i),y(i),θ n.(3.7) Then, to apply the gradient descent method is needed to calculate the gradient with respect to the parameters of the model θas ∇θL(θ) = ∑n i=1∇θLx(i),y(i),θ n.(3.8) Computing the gradient descent with this calculation is one of the most successful options. However, as we commented above, the size of the dataset is huge and makes difficult training with the whole data due to memory space restrictions. The solution for that is training using a subset of data which is called batch size, and can be defined with any desired size. For the cases of really small batches they are called minibatches and for some fields they shown better results that bigger batches. Denoting the loss for these batches as L0and the number of elements in the batch as n0the gradient could be defined as be ∇θL0(θ) = ∑n0 i=1∇θLx(i),y(i),θ n0.(3.9) Finally, the gradient descent for the parameters using batches is θt=θt−1−ε∇θL0(θt−1).(3.10) This variation of the gradient descent is called stochastic gradient descent and is one of the most used methods in the training of neural networks. As the original method, this one can not converge to the global minimum but, it will end in the nearest local minimum. 3.2 Non-line-of-sight (NLOS) transient imaging Transient imaging methods leverage the information encoded in the temporal domain of a timeresolved light capture. One of the areas that can be addressed in this field is the non-line-ofsight (NLOS) reconstruction. The imaging methods developed in this area aim to reconstruct scenarios that are hidden around a corner, by analyzing their indirect illumination on a surface visible to the camera. A typical NLOS scene can be seen in figure 3.2 where an ultra-short laser pulse is emitted to a wall (also called relay wall), the light propagates from the hidden scene, being reflected by the objects, and part of it goes to the relay wall and is recorded by an ultra-high speed camera called single-photon avalanche diode (SPAD) generating a transient profile Hcalled impulse response function. These profiles have the information of the hidden A machine learning approach for transient imaging reconstruction 13 scene encoded inside them. Thus, His calculated for different points of the relay wall for the laser and the SPAD parameters are H(xl,xs,t)where xlis the laser position, xsis the sensor position and ta specific temporal instant. NLOS SPAD-laser capture setup SPAD laser Figure 3.2: Sample setup for capturing a NLOS scene. An ultrashort light pulse is emitted to a relay wall and the sensor captures the light resolved in time reflected by the scene to the relay wall. Image adapted from Liu et al.[2]. The impulse response function His used in several methods for reconstructing the hidden scene as we commented in section 2.1. In this work we focus in the phasor field framework [2] which is one of the best performing methods in NLOS imaging. This work gives us the possibility to transform the Hfunction into a phasor field which transforms the temporal resolution into a complex magnitude at the relay wall with amplitude and phase dimensions for a concrete frequency. 3.2.1 Phasor fields The work of Liu et al. [2] presents a new method for transient imaging which transforms the NLOS problem into a virtual line-of-sight (LOS) problem. This transformation allows the authors to use classical optic methods in the NLOS domain. Virtual illumination Virtual aperture Virtual sensor Virtual lens (a) (b) (c) Figure 3.3: Phasor field steps. (a) Virtual illumination is propagated through the scene. (b) A virtual aperture capture the response of the scene to the virtual light. (c) A virtual lens focus the captured illumination of the virtual aperture and imaging. Image adapted from [2]. 14 Chapter 3. Background The linearity of H(xl,xs,t)is used by phasor fields to compute the response of the hidden scene (Fig. 3.3b) for any virtual complex-valued emission profile P(xl,t)(Fig. 3.3a) at points xlin a virtual sensor as P(xs,t) = ZL [P(xl,t)∗H(xl,xs,t)]dxl.(3.11) Through the propagation of the field P(xs,t)with an imaging operator I(·)for any point xvin the hidden scene as I(xv) = Φ(P(xs,t)).(3.12) This operator models a virtual lens and sensor system (Fig. 3.3c) and it can be formulated in terms of a Rayleigh-Sommerfeld diffraction propagator [2]. In the case of propagating monochromatic signals of a single frequency ωthis image formation operator Φ(Pω(xs,t)) is denoted as Φ(Pω(xs,t)) = ZS Pω(xs,t)Lω(xs,xv) |xv−xs|dxs 2 ,(3.13) where Lis a complex operator that changes the phase of Pat frequency ω. For the Φoperator we implement it as a thin lens model that focuses into a virtual image plane at a hidden location xv, having Lω(xs,xv) = e−ik|xv−xs|,(3.14) where k=ω/cis the wavenumber, with c the speed of light. A position xvcan be imaging by combining the focused emission profile (Eq. 3.11) and the imaging operator (Eq. 3.13) as I(xv) = ZSZL [Pω(xl,t)∗H(xl,xs,t)] Lω(xs,xv) |xv−xs|dxldxs 2 .(3.15) This method is fully computational since the unique external data that it need is the H function. Furthermore, the image formation model is virtual and can be formulated in other forms. The same idea holds for the emission profile and for this work we will define it as a constant-emission light source as Pω(xl,t) = eiωt.(3.16) Chapter 4 Implicit representation of phasor fields In the last decade, the field of non-line-of-sight (NLOS) has evolved improving the quality of the reconstructions of hidden scenes. However, the capture method has barely changed. NLOS methods use a captured impulse response function of a hidden scene. To obtain this function a light is emitted, by an ultra-short laser, to a diffuse wall (relay wall). Then it travels across the hidden scene and goes back to the relay wall where it is finally captured by a sensor. The capture process needs to be done sequentially for each point in the relay wall, resulting in a slow process with a bottleneck that forces to have impulses response functions with low spatial resolution. Despite the problem with the spatial resolution, works as Liu et al. [2] successfully reconstruct high complexity hidden scenes. However, to obtain more detailed reconstructions, it is mandatory to generate more spatially dense impulse response functions. To obtain them there are two options: creating new hardware to increase the spatial resolution or using computational methods to generate new points from the captured ones. On the one hand creating hardware to increase the spatial resolution of the sensor is not a trivial work. The actual sensors are single-photon avalanche diode (SPAD). They have a resolution of 1x1 pixel and there are some versions with a line of SPADs obtaining a row of captures. Despite these advances, the next generations of 2D SPADs will have a low resolution near 16x16 pixels which is insufficient to solve the problem. On the other hand, using fully computational methods can work with the actual capture systems. We can test and implement multiple algorithms or versions of them in a relatively short space of time. In spite of the advantages that computational methods can give us, the main limitation for us to design and create new capture hardware is the necessity of testing it with real capture systems which only a few laboratories in the world have. In contrast to test computational methods we can use both real and simulated data, and therefore, validate the method by using them. For that reason we have decided to implement a computational method and generate, through simulation, our capture data. The problem that we want to alleviate is the low density of values in the spatial domain of the impulse response function to improve the quality of the reconstructions. We can say that this problem can be understood as an upsampling or superresolution problem where the number of samples of a function is increased by an algorithm. This problem is typical in images where the low resolution image is transformed into a high resolution one. The type of methods that gives better results in this domain are the machine learning based ones. However, to work with 15 22 Chapter 4. Implicit representation of phasor fields Figure 4.5: Example of Hfunction for a concrete pair laser-sensor. The top image is the transient profile in its original scale. The bottom image is the transient profile in logarithmic scale. To solve the problem with the high dynamic range of the Hfunction, we propose to use the phasor fields framework (section 3.2.1) for NLOS reconstruction. In the first step of this framework the Hfunction is convolved (eq. 3.11) with a virtual illumination phasor for a concrete frequency ω(eq. 3.16). This phasor P(xs,t)is a 3D complex matrix, two spatial dimensions and one temporal, but as the virtual light used in the convolution is constant along the time and the camera model is a conventional camera, P(xs,t)can be evaluated in P(xs,t=0)obtaining a 2D complex matrix and solving the problem with the high dynamic range. However, the frequency that modules the virtual illumination affects to the reconstruction quality as can be seen in figure 4.6. Furthermore, the correct frequency for each Hfunction is not the same and needs to be obtained experimentally. Frequency Hidden scene +_ Figure 4.6: Example of different reconstructions with different frequencies. Data normalization. In this work we have two types of data: the input parameters of the model (coordinates and in some models the frequency) and the values of the phasor which the model will learn. For both options we tested the [0,1]and [−1,1]normalization but the models A machine learning approach for transient imaging reconstruction 23 did not fit correctly in the borders of the matrix. Since the problem was in the boundaries we consider that the problem was with the normalization values in the limits of the range and with a normalization in (−1,1)this problem was palliated. Data generation. The capture of the Hfunctions can be done from two different sources, real capture systems or through simulation. The first one can not be done in the laboratory at this university due to we do not have the hardware. However, using real captured data have some disadvantages. The captures have noise and their conditions can not be controlled completely. On the other hand, simulating avoids this problems due to the environment can be controlled (e.g., avoid external light contamination) and can be generated with less noise. Also, simulating allow to create idyllic conditions e.g., perfect lambertian surfaces for the relay wall. For that reason in this work we simulate all the captures using the public transient render by Jarabo et al. [13]. Generic model vs specific models. Neural networks are usually applied for generic applications that can be used once is trained with different data, for example in superresolution models, classification or some compression works. However, this models need to be trained with enormous datasets, in some cases millions of samples. In the context of this work this is not an option due to the amount of time that is needed to generate a dataset of the size that general models need. For that reason in this work we choose to use one model per function (or scene) that we want to fit. Since we are fitting a single function in the neural model, and we want to infer new values, we need to avoid overfitting in the model. In other cases, if we wanted to completely compress a function and only evaluate at the original points, overfitting would be justified. Other aspect that is involved in the overfitting and in the capacity of the neural models to be able to fit the discrete function is the number of model parameters. The correct number of them needs to be obtained experimentally. Moreover, it can depend on the type of function that the model is fitting, complex functions can require more parameters than the simplest ones and if the number of parameters is too big, the model will be lazy and will overfit easily. In the context of this problem, we can consider an upper bound for the number of parameters, the total amount of values in the discrete function because it is the number of values needed to represent the same information. In our experiments we tested with different number of parameters and figured out that with more than half the total number of parameters of the original function, the models tend to overfit and can not generate well new points. On the other hand, with less than the half the models learn the discrete function and are able to generate new points smoothly. However, if the parameters are too low, the models can not converge at all. 24 Chapter 4. Implicit representation of phasor fields Chapter 5 Results In the section 4.1 we have tested different neural networks models to learn an implicit representation of a phasor field. Specifically, the neural network model obtained can learn an implicit representation of a multi-frequency phasor field. One of the advantages of learning a multi-frequency phasor field is the elimination of the necessity of calculating separate implicit representations for each frequency. This means that it gives the ability to compute any desired frequency that it is not in the sampled space. This is especially interesting due to the frequency is directly dependent of the quality of the reconstructions. Computing novel phasor fields with higher frequencies and higher grid densities can provide sharper results of the reconstructed geometry. As such, the ability of our implicit representation to compute novel temporal frequencies and spatial locations in the relay wall that were not provided by the sampled captured data is useful to provide better reconstructions. Another application of this multi-frequency model is the possibility to use it with more complex lights and cameras in the phasor field framework. To use them it will be required (theoretically) the use of infinite frequencies. With an implicit representation, this problem can be more treatable because the representation avoids the necessity to do any convolution for each frequency. During the analysis of the different neural models (Section 4) we have tested them with only one scene for simplicity. After demonstrating that the MLP with sine activation functions model provides the best behavior in representing multi-frequency phasor fields, here we provide a deeper analysis of its performance in scenes of different complexity. However, before showing the different results, is important to understand the limitations of the virtual camera and the virtual light used in the phasor field framework. The virtual camera has an aperture size that corresponds to the size of the sampled grid on the relay wall. After using this aperture to simulate a virtual camera focused at certain depth in the hidden scene, all geometry behind or ahead of the focused plane is out of focus, introducing several artifacts in the image. Additionally, due to capture limitations, the virtual illumination is attenuated radially from the center of the relay wall, which results in attenuated reconstructions of the geometry further from the center of the resulting images. This effect is clearly visible in all the previous results in the section 4.1. To analyze how the implicit representations can approximate the discrete phasor field we will compare the intensity, the phase and the reconstruction of both phasor fields, discrete (ground truth) and the one predicted by the implicit representation. We have selected simple scenes to have low out-of-focus geometry to avoid out-of-focus artifacts. Reconstructing the in-focus geometry in more complex scenarios would require estimating a large amount of different phasor fields for each voxel of the reconstructed scene, while simple scenes provide good results with a single multi-frequency phasor field. We regard as future work to provide 25 26 Chapter 5. Results more complex implicit representations for phasor fields in cluttered scenes. As in the last model of the section 4.1, we will test the capacity of the model to generate new samples in the spatial and frequency domain. To train and test the model we have decided to remove 75% of the samples from a phasor field with a spatial resolution of 64x64, therefore getting only 32x32 sampled points in the relay wall. For the frequency, we have computed 151 different frequencies and trained with the 90% of them, reserving the last 10% for testing. Once the model is trained with the low resolution phasor field, we recover the original resolution at specific frequencies and spatial locations in the relay wall that the model has not been trained with. For testing the model, a multilayer perceptron with sine activation functions, we have used four different scenes (figures 5.1-5.4). We start by a planar but detailed scene (figure 5.1). This scene allows us to analyze the performance under different geometric resolutions allowing seeing the level of details that the method is able to recover. Concretely, the results obtained for this scene show the capability of the model to learn more complex phasor fields and upsample them correctly.The general structure is maintained and an important denoised effect is appreciable. The reconstructions are qualitatively similar. They lose a bit of detail but, bearing in mind that the model is trained with the 25% of the original points, it is a good reconstruction. As comparison, in the figure 5.2 can be seen how is a reconstruction of a phasor field with a resolution of 32x32 and a reconstruction with a resolution of 64x64. In the error map it can be seen two things: the error derived from the noise not learned by the model and how the network fails more in the low values. Figure 5.1: Results of recovering a scene with multiple planes with different sizes. The results show that the implicit representation learn correctly the shape of the phasor fields and remove the noise of them. The reconstructions are qualitatively similar. Left phasor field has a frequency of 11.422 and the rigth one has a frequency of 15,238. The second scene is formed by a plane with several concavities and irregularities (figure 5.3). Note how the resulting amplitude and phase of the phasor field change due to the structural differences of the hidden geometry. Nevertheless, our implicit representation is able to estimate the phasor field structure at higher resolutions, very close to the ground truth while also removing noise, and provides a similar reconstruction result. Our implicit representation A machine learning approach for transient imaging reconstruction 27 Figure 5.2: Comparison of the reconstruction of the same scene with different resolution, 32x32 and 64x64 for the same frequency (15.238). is capable of learning the general structure, while simultaneously removing the noise in the amplitude and phase images. The error maps show, as in the previous example, that the model has more error in the low value points and also the general noise error. The reconstructions are very similar. Despite that the RMSE in the first reconstruction is higher, the result is qualitatively more similar than in the second reconstruction. Figure 5.3: Results for a scene with multiple concavities and pikes. The reconstructions do not show the shape correctly due to the out-of-focus geometry. The neural model learns and recover the shape of the phasor field removing the noise from it. Finally, we have two similar scenes (figures 5.4 and 5.5). Both are formed by two rectangular planes at two different depths but in each scene the distance between them is different. The goal of these two scenes is to test the performance of our algorithm under the presence of objects that may occlude themselves from the perspective of the virtual camera. For these two cases our implicit neural model learns the structure of the phasor fields as well. Comments and discussions. Our neural representation of phasor fields has shown a high capacity to learn the general structure of the phasor field, being able to generate higher resolutions (recovering a 75% of the points in these tests) and new frequencies with similar results to the ground truth. The RSME is similar in all of them but is bigger in the phasor field with higher frequency. This could be caused by the increase of details that appear when the temporal modulation frequency is increased. This increase in the temporal frequency results in 28 Chapter 5. Results Figure 5.4: Results of a scene with two planes at different depth. This scene allows us to test if the phasor field with multiple objects and depths can be learned by the neural model. The model learns correctly despite the changes in the shape. Figure 5.5: Results of a scene with two planes at different depth. In this case the plane in the back is semioccluded by the plane in the front. The neural model learns correctly the shape and even the strong change of values. an increment of the spatial frequency in the phasor field, as the results show. Focusing on the reconstructions, the predicted and the ground truth are also very similar and can recover even the artifacts produced by diffraction effects. Note that this is a problem inherent in the phasor fields method [2], and not a result of our neural model. The biggest qualitative differences between predicted and ground truth appear in the reconstructions done with the phasor fields with higher frequencies, losing more details than with the low frequency phasor fields. Note that our quantitative comparisons are done with respect to the ground truth solution with equivalent sampling density and temporal frequency. While our model yields a larger RMSE when increasing modulation frequency, increasing sampling density and modulation frequency in phasor fields provides better reconstructions of the hidden scene, as our qualitative results show. The differences in RMSE of our model with respect to lower frequencies arise from the ability of our model to analytically represent higher frequencies, and not from the ability of the data to properly reconstruct the scene. A machine learning approach for transient imaging reconstruction 29 Hidden scene Loss (L2) Training time (s) Figure 5.1 0.009883426 677.92 Figure 5.3 0.011194097 670.95 Figure 5.5 0.0029691448 673.33 Figure 5.4 0.0032978356 673.98 Table 5.1: Losses and training times for each scene. The training process have been done with a learning rate of 0.001 and a total of 100000 epochs. The model in all our tests has shown a denoising ability. This effect is produced by the capability of neural networks to focus on predictable aspects of the data. Concretely, as the noise is something more random than the structure of the data, the neural networks obtain better results learning the structure and characteristics of the data. However, this effect is only observed when the parameters of the networks are not too high because, in that case, the neural network overfits the data and cannot upsample. Regarding the possible effect of the noise in the error maps, we thought that it is possible that the noise could mask the correct level of achievement of this method. However, to test it, it would be necessary to use other metrics that take more into account the structure of the nearby pixels as the structural similarity index measure (SSIM). We will leave the test of this hypothesis for a future work. In the reconstructions, the effect of the existence of noise in the phasor field does not appear to be determinant or at least the effect is too low to be seen. In their error maps, the first difference regarding the error maps of the phasor field is the lack of noise. The error maps of the reconstructions are more smoothly than the error maps of the phasor fields. We thought that the reconstruction process is resistant to this level of noise. With this idea, a future work that could be done is an analysis of the tolerance of the reconstruction method at different levels of noise and how this method can be used for denoising it. This is an interesting aspect because the real captures have noise that usually needs to be added to the synthetic data to compare both. Furthermore, knowing that, it could help to know the level of noise that this method can handle and use it to generate synthetic data faster. The training process of these scenes is fast (see table 5.1 for exact training times). In around eleven minutes each scene can be learned. The evaluation process for each frequency is practically instant, less than one second, giving a huge potential to use it for more applications such as using more complex cameras that could require computing thousands of different frequencies. Moreover, the model can be evaluated with multiple frequencies at the same time without a perceptible increase of time. 30 Chapter 5. Results Chapter 6 Conclusions and future work In this work we have introduced a new method to represent phasor fields in an implicit form by using neural models to increment the efficiency and the quality of the results of reconstructions of hidden scenes. Phasor fields are a recent work that allows to take virtual photographs of a hidden scene as it was seen from a relay wall. However, this method is limited by in its resolution caused by the capture process. Since each spatial point has to be sequentially captured, it becomes unpractical and complex to capture high resolution information. Although this problem could be alleviated with the development of new hardware, this approach would require the use of expensive hardware configurations. Inspired by other works on implicit representations that prove their ability to create a continuous space from discrete number of samples. We have formulated a phasor field using implicit representation which allows us to learn a discrete phasor field and sample it in a continuous space. This transformation removes any resolution limitation. In other words, we could obtain infinite samples from it. Following recent studies and works, we have tested different models for different tasks. Starting by the most simple (learning an implicit representation for a singlefrequency phasor field without changing the resolution), up to the most complex (that learns an implicit representation of a multi-frequency phasor field). This last model is a multilayer perceptron with sine activation functions that can upsample the data in the spatial and frequency domains. To verify if the final model can generalize to different scenes we have tested it with four scenes. The model correctly upsample the data for all the scenes giving a similar error in all of them. Other aspect observed is the denoising effect, since the model recovers smooth phasor fields with the correct structure. This can be qualitatively demonstrated, as the reconstructed scenes are almost identical to the ground-truth ones. This work left some interesting research as future avenues. Our current method could be extended to support more cluttered scenes with more complex imaging functions, where each location of the scene would require a different illumination and lens function, leading to the estimation of higher number of phasor fields per scene. An aspect that can be tested is whether the noise in the captures is related with the differences in the reconstructions. Related to the noise, it could be interesting to analyze how the noise affects to the reconstructions and if this model or other types of network can denoise the phasor field and with which level of noise are they effective. Moreover, this work could be expanded to the study of the whole four dimensions involved in phasor fields, in contrast to the two explored in this work. Another 31