A Prediction Model for Neuronal Synaptic Inputs

Bachelor’s Degree Final Thesis B. Tech. in Industrial Technology Engineering A Prediction Model for Neuronal Synaptic Inputs MANUSCRIPT June 21, 2020 Author: Pau Fisco Compte Supervisors: Enric Fossas Colet Nestor Roqueiro Convocation: 02/2020 Escola Tècnica Superior d’Enginyeria Industrial de Barcelona A Prediction Model for Neuronal Synaptic Inputs page 1 Abstract In this manuscript it is exposed a method to approximate functions using artificial neural networks based on wavelets (wavenet). The focus is on finding the best configuration for the wavenet, from various possible settings in relation to the mathematical development of the network, to be able to approximate a FitzHugh-Nagumo model to later be used to predict any scenario with a single initial condition for the model. It is shown that after training the artificial network, it is able to approximate the non-linear behaviour of the FitzHugh-Nagumo model with high accuracy, providing a neuron model which can be then applied to models for real neurons with similar inputs to those from the wavenet. Finally, it is proved that additional linear terms applied to the outputs improve significantly the error of the approximation to those wavenets with non-linear type scale functions, thus it is been possible to obtain better results without the need to increase the resolution level and thereby reducing the simulation time. page 2 Manuscript Contents Abstract 1 1 Preface 5 1.1 Project’s origin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Previous requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Introduction 6 2.1 Project objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Points included . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3 Approach to neural networks 8 4 Mallat’s multiresolution analysis 10 5 Approach to wavelets 12 6 Wavenets 14 6.1 Unidimensional function prediction . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.2 Multivariable function prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 6.3 Linear relations and offset parameter . . . . . . . . . . . . . . . . . . . . . . . . . . 16 6.4 Exponential wavenet’s growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 7 Loss function. Mean squared error 19 7.1 Mean squared error regularizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 8 FitzHugh-Nagumo model prediction 21 8.1 Three inputs case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 8.2 Activation function superposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 8.3 Optimal wavelons’ configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 8.4 Effect of the linear terms in the error . . . . . . . . . . . . . . . . . . . . . . . . . . 26 8.5 Final results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 9 Economic balance 30 10 Environmental Impact 31 11 Conclusions 32 12 References 33 A Prediction Model for Neuronal Synaptic Inputs page 3 page 4 Manuscript A Prediction Model for Neuronal Synaptic Inputs page 5 1 Preface Computational neuroscience is the theoretical study of the brain to uncover the principles and mechanisms that guide the development, organization, information processing, and cognitive abilities of the nervous system [1]. The general paradigm in computational neuroscience is focused in the study of neurons, and using that knowledge one can design an application that can solve specific problems according to the functionality of these cells. For example, a grasshopper neuron that processes images [2], for any computer vision applications. This scenario generates the necessity to study different neurons, probably from different kinds of simple animals, to create any desired application. In engineering, all those specific neurons are pretended to be obviated and the goal is to develop one model or few, to create generic algorithms for any kind of applications. 1.1 Project’s origin This manuscript surges from a previous work [3], in which the Morris-Lecar (ML) model is used to describe the behaviour of a neuron. The model’s goal is to create data for a neural network (NN) based on wavelets, a wavenet, without the need to know the intrinsic properties of the neuron, its morphology, the ionic channels, the neurotransmitter availability, etc. The challenge of the present project, as in the referred previous work, consists on being able to predict the behaviour of an excitable system, like one described by a neuron, considering it as a black box, thanks to the use of a wavenet, using the FitzHugh-Nagumo (FHN) model instead of the ML worked on [3]. 1.2 Motivation The growing interest in artificial neural networks and the control over Python have been a motivation to try to reproduce similar work done in [3]. If the inputs of a neuron can be predicted using a NN, it is possible to predict more complex neurons making variations of the model with the same kind of wavenets. With this work it would be possible to create a wavenet that can be used for complex industrial processes [4], because these can be difficult to describe due to the amount of variables interacting. One of the jobs of an Industrial engineer is to optimize processes, and one way to optimize nonlinear complex systems is with the use of NNs and artificial intelligence. Many applications have focused in this direction and this thesis is an introduction to what the future has to offer. 1.3 Previous requirements To be able to make this project, it is necessary to have basic knowledge about Python3, due that the wavenet will be written in this programming language. It is necessary to know how to implement certain scientific libraries as numpy and graphic libraries such as matplotlib. It is also necessary to have a background about ordinary differential equations (odes), as this kind of equations will be the ones conforming the mathematical models, describing the neuron behaviour (like the ML and FHN models), to be able to train the wavenet. The Euler’s method will be used to solve these equations, to compare the analytical solution of the functions with the predicted values from the NN. page 6 Manuscript 2 Introduction This document diverges from the tools used in previous works in which this project is based. To create the NN, Python is directly used from the equations behind the wavelet’s theory. These equations have been developed and adapted to satisfy the needs of the wavenet. It is created without specific tools for this purpose or any tool used in NNs, like the ones offered in Tensorflow or Pytorch. All wavenet’s routines have been integrally developed. For this reason, the optimization of the code also has been an objective, so the training time can be reduced and changes in the wavelet’s configuration can be applied easily. Starting from the work accomplished in [3], there is the possibility to obtain real data from directly measured voltages from the neuron of a giant barnacle (following the ML model) and to test the behaviour of the wavenet in a real world scenario. As seen in [3], the model works successfully, and in this project it is presented the same goal but with the FHN model. First of all, it is important to get used to wavelets and the different types of NNs, to know which are the ones that fit the purpose of the project. Once implemented these equations to the NN, it will get more complex by increasing the number of inputs and adding linear terms and an offset, if it is necessary. 2.1 Project objectives The principal objective of this project is to create a wavenet, trained with a grid of initial conditions, large enough as to contain all possible outputs of the FHN model in front of any random stimulation. The objective can be divided in more simple ones: •To understand the concepts about the theory of wavelets and how to use them to predict functions. •To know the different topologies of a NN, and the one fitting a wavenet, that is the number of neurons and the relations among them. •To find a way to obtain the predicted function in a fast and precise form, making use of the theory around wavelets and any possible Python programming technique, to acknowledge the best structure configuration for the wavenet, focusing on the goal to obtain a minimum training time and a minimum error. 2.2 Points included With a three months investigation and development of the NN using Python, the phases of this document are: •To search for the possible structures for the wavenet and the different resolution methods to find the predicted function. •To develop algorithms that satisfy the correct use of neurons for the type of problem. •To study ways to optimize the Python code to accelerate the training by making it faster and by reducing the amount of calculations needed to obtain a wavenet with less complexity with the same accurate result. A Prediction Model for Neuronal Synaptic Inputs page 7 •To do an economic analysis to study the project’s cost and its environmental impact. page 14 Manuscript 6 Wavenets The use of sigmoid functions in typical NNs might converge in a local minimum using a training algorithm. Trying to keep it simple, instead of a rough feedforward NN, wavenets are a variant of them and solve the problem of local minimum finding always the global minimum solution. A wavenet (WN) is a generalization of a radial base function network (RBFN). The RBFN are actually feedforward NN that use radial base functions, like wavelets, as activation functions instead of logistic functions, like sigmoids. Knowing this, the basic structure of the WN is just one hidden layer apart from the input and output layers. The nodes, or so called wavelons of the WN, are vectors of wavelets, as activation functions, applied on the inputs. Every single wavelon is multiplied to a weigh, which corresponds to the coefficients of the function expansion seen in the expression (4.6), also known as the wavelet base to a certain subspace Vj. Having that said, if a WN learns how to approximate any function following the expression (4.6), the WN is always applying Mallat’s multiresolution analysis to do so. By using radial base functions, the result only depends on the distance from the origin of one variable. Computing the value of the input at a single point in the wavelon or updating the estimated function from a new local measure involves only a small subset of coefficients [7,11], thanks to the orthogonality property, if not, every single time the estimated function is updated all the coefficients should be calculated at once. The output, following a series expansion, is generally expressed by the equation (4.6). This is why the hidden layer grows in each update of the estimated function, and every update corresponds to a level in the sum from the same expression, determined by the level of resolution from the Mallat’s multiresolution analysis. WNs also provide information for the relative participation of each wavelon to the function approximation and the estimated dynamics of the generating process. 6.1 Unidimensional function prediction In this part of the project’s chapter, it will be exposed the simplest system for a WN. Knowing that the number of mother wavelets is computed as 2E−1, where Ecorresponds to the number of inputs given to the WN, it is possible to know the number of mother wavelets, or simply called wavelets, that perform the orthogonal base with a single scale function. For one input, the predicted function depends only on one dimension, and can be expressed by the equation (4.6), particularized in the form seen in (6.1). f(x) = dφ(x) + M X m=0 N X n=0 cm,nψm,n(x)(6.1) Where, the new parameters seen in the expression mean: M: maximum level of resolution. N: maximum translation of the rank 2mwavelet where its value is not null. Using the mother wavelet equation (5.2), with the defined values a0and b0, the mother wavelet A Prediction Model for Neuronal Synaptic Inputs page 15 can be seen in (6.2). ψm,n(x) = 2m/2ψ(2mx−n)(6.2) Parameter nwill take values depending on the rank of the wavelet, from zero to N, where N=M−1, with Mbeing the rank, corresponding to 2mfor the resolution m. The wavelet’s input is an array of the different values from the variable that will be computed in the scale function and the wavelet. Defining the input variable as x, there will be iindices in the array, for i= 1,2...X. With this notation, a model ˆycan be expressed as a matrix system given by the following equation (6.3). ˆy(x) = F(x)σ(6.3) The σvector corresponds to the calculated weights for the WN, and finding the best model that predicts the f(x)will depend on the solution for this vector. The generic dimensions of (6.3) as a matrix form are seen in table 6.1. Table 6.1Matrix dimensions. Matrix Dimension ˆy X x 1 F X xJ σ J x 1 Where Jcorresponds to the growth rate of the wavelons. J= 1+PM m=0 2min a unidimensional WN. 6.2 Multivariable function prediction Here it will be exposed a bidimensional WN, to simplify the characterization, but once a wavelet base is defined and the growth rate of the WN is set, any multivariable function can be approximated without effort with enough computational power. To know how many wavelets there are in the wavelet base, using the expression at the beginning of chapter 6.1, for a two-input case, it is set to three wavelets. The wavelet family will have the following form, seen in (6.4). ψh m,n(x1, x2)=2mψ(2mx1−n, 2mx2−n)h= 1,2,3(6.4) The 2mcoefficient multiplying the wavelets here, as the 2m/2coefficient seen in (6.2) are meant to normalize the base, to make it orthonormal, but it does not matter for the goals of this project as long as the base is orthogonal. The coefficients on the σvector will take the required value to approximate the given function and the norm will be implicit in that value. The scale function will also be bidimensional, and using the same scale function for every input variable gives an advantage to simplify the calculus, as the scale function will have the expression in (6.5). φ(x1, x2) = φ(x1)φ(x2)(6.5) page 16 Manuscript If the scale function is a combination of more than one scale function, here takes significance the first sum term in the expression (4.6), where now the function φ(x1, x2)has the following general expression (6.6). φ(x1, x2) = Nφ X j=1 Nφ X i=1 φi(x1)φj(x2)(6.6) Where: Nφ: is the total number of scale functions. And characterizing the expression for a two-functions case, the specific base expression results in a vector seen in (6.7). Φ(x1, x2)=[φ1(x1)φ1(x2), φ1(x1)φ2(x2), φ2(x1)φ1(x2), φ2(x1)φ2(x2)] (6.7) To make it simple, the following expressions will be focused in a single scale function case. Using the (6.5) expression, the mother wavelets from the wavelets’ family seen in (6.4), are expressed as a base in (6.8). Ψ(x1, x2) = [φ(x1)ψ(x2), ψ(x1)φ(x2), ψ(x1)ψ(x2)] (6.8) The growth rate of the WN, set by the number of wavelons needed in every resolution level is expressed as seen in the equation (6.9). Wavelons = 1 + M X m=0 (2 ·2m+ 22m)(6.9) Finally, it can be generalized a formula, as a sum of terms, for the model ˆyin a bidimensional WN, as seen in (6.10). ˆy(x) = dφ(x1)φ(x2) + M X m=0 ( N0 X i=1 N X n=0 cm,iφ(x1)ψm,n(x2)+ + 2N0 X i=N0+1 N X n=0 cm,iφ(x2)ψm,n(x1)+ (6.10) + 2N0+N02 X i=2N0+1 N X l=0 N X k=0 cm,iψm,l(x1)ψm,k(x2)) Where N0= 2m. The matrix system will follow table 6.1 taking into account the wavelons’ growth changes seen in the formula (6.9) and the development seen in (6.10). 6.3 Linear relations and offset parameter In a WN, the output is established as a sum of linear relations among the values of the wavelons, and these linear coefficients are determined by the weights located between the hidden layer and the output layer, as explained in previous chapters. It can be introduced two new types of weights that are not treated as the weights associated to each wavelon. A Prediction Model for Neuronal Synaptic Inputs page 17 The firsttype of weight corresponds to a direct linear relation between any input and the output. To apply this relation in the WN, the normalized data going into the wavelons is multiplied directly to a weight and the result is summed to the predicted output, it is done by adding a column of normalized inputs in the wavelet matrix F(x). The second type of weight is just a constant value, called offset, applied to the output. There is the option of getting an independent weight to the output by adding a column of ones in the wavelet matrix F(x), and then multiplying the vector to a weight. It can also be used a haar scale function, seen in table 4-1, with the normalized inputs, the haar will take value 1 and the weight will be calculated. Figure 6-1. Linear term represented as the weight w1 1and the offset term as the weight w0 1. The graphic representation of the linear relation and the offset is shown in figure 6-1, following a similar WN structure found in [11]. 6.4 Exponential wavenet’s growth The main clue in regards to building a WN is to know exactly its growth in every resolution’s level. As seen in Jfrom table 6.1 and the expression (6.9), the WN’s growth is a linear relation among a set of exponentials determined by the number of inputs the WN has. Empirically, it can be found a general expression for the variation of WN’s wavelons, according to the number of inputs and the level of resolution, as seen in (6.11). Wavelons = 1 + M X m=0 E X i=1 Ki2im (6.11) Where: M: maximum level of resolution. E: number of inputs K: multiplicity constant that represents the number of wavelets from the same mother wavelet for each iinput, defined in the second level of resolution. page 18 Manuscript The expression is followed by three rules: 1st rule: K= 1 if i=E 2nd rule: Ki1=Ki2where i1+i2=E 3rd rule: PE i=1 Ki= 2E−1 It can be defined a generic second level of resolution, corresponding to the first orthogonal base, to easily find all Kconstants from (6.11) thanks to these rule,s and extrapolate the WN’s growth through each level doing a minimum amount of operations. Doing so, it is possible to create table 6.2 rapidly. Table 6.2 – WN’s growth according to different number of inputs. The independent term in (6.11) corresponds to the single wavelon with the scale function as the activation function, then it is added any of the rows from the table above inside the summation with the maximum level of resolution as the upped bound of summation. From a computational perspective, it is important to know how it behaves the exponentials’ set in a WN. In the developed algorithm, every exponential corresponds to a for loop and every K constant matches the number of new variables created, in each one of these loops, to save the new vectors which will be added to the wavelet matrix F(x). That is, with the algorithm used to create the WN, if it is necessary to modify the number of inputs, it will be no hard work and the major part to be changed in the code is defined by the WN’s growth behaviour expressed in (6.11) and the rules it follows. A Prediction Model for Neuronal Synaptic Inputs page 19 7 Loss function. Mean squared error Once the WN’s structure is set, it has to be trained to obtain the best solution for the weights. With the matrix system obtained from the WN, similar to the one in (6.3). The mean squared error (MSE), as an objective function, will measure the quality of the predicted model ˆyand it is used for regression tasks, like linear regressions. This loss will calculate the squared differences between the target and the model. Therefore, applying the mean, the more data is added the less error is yield. The form of the MSE for matrix systems is a multiple linear regression expressed in (7.1). MSE =1 N y(x)−ˆy(x)  2=1 N y(x)−F(x)σ  2(7.1) If the previous expression is developed it becomes (7.2). MSE =1 N(y(x)|y(x)−2y(x)|F(x)σ+σ|F(x)|F(x)σ)(7.2) As the MSE is the function to minimize to find the optimal solution for σ, the optimal condition will be ∂MSE ∂σ = 0. Applying the product transposition property from matrices, the expression to solve is (7.3). ∂MSE ∂σ =−2y(x)|F(x) + σ|F(x)|F(x)(7.3) Which can be finally expressed as (7.4). F(x)|y(x) = F(x)|F(x)σ(7.4) Being a new system in the Ax =Bform, where Ais the covariance matrix F(x)|F(x)and B the product of the transposed wavelet matrix F(x)|by the target matrix y. As it is shown in (7.4), the formula is the same as the one used to solve the system (6.3) by least squares and it could have been the way to go since the beginning because it is a low rank system, as the wavelet matrix will have more rows than columns. Instead of using least squares, it is often used the backpropagation algorithm in a feedforward NN to minimize the loss function, but it can converge to a local minimum, whereas the least squares solution is a global minimum. 7.1 Mean squared error regularizer Following the studies conducted in [10], it is introduced a new term, a regularizer, in the objective function and as a result it becomes a regularized objective function. The meaning of the regularizer is to be able to smooth the curvature of the model’s surface by reducing the oscillations around the data used as a target for the model, providing a great scalability. The regularizer also optimizes the computational calculus avoiding the possible problems related to the inverse covariance matrix when solving the system in (7.4) to find the weight’s vector. The maximum and minimum eigenvalues from the covariance matrix tend to zero and this drives an uncontrolled growth of the number of conditions given by the expression λmax λmin . If the λmin tends to zero the estimated weight’s vector can be inaccurate due to the inverse. page 20 Manuscript The simplest regularizer is the norm vector of the squared weightskσk2, giving the new loss function (7.5). G= y(x)−F(x)σ  2+γkσk2(7.5) Where: G: is the new loss function γ: is a multiplier Following the same methodology as with the MSE, the new optimal condition is ∂G ∂σ = 0. The new expression to solve the weight vector is shown in (7.6). F(x)|y(x)=(F(x)|F(x) + γI)σ(7.6) With the (7.6) formula, it can be observed the Ax =Bform as well, but the single difference from a non-regularized loss function is the identity of multipliers in the Amatrix, added to the covariance matrix. This way, it can be said that the equation in (7.6) is solved by least squares. The expression of γis given by (7.7). γ=µλmax (7.7) Where: λmax: corresponds to the maximum eigenvalue from the covariance matrix. µ: is a constant to be found. The value µdepends on the application and, as it is said in [10], the numeric stability from an inverse of a matrix is defined by the number of conditions related to the constant with the expression 1 µ, so µ=λmin λmax . For the FHN model it will be used the value µ= 1e−18, which has given the best results. A Prediction Model for Neuronal Synaptic Inputs page 21 8 FitzHugh-Nagumo model prediction The main objective of this project, as it is already said, is to be able to train a NN to perform as a real neuron. There are many models that can describe the behaviour of a neuron, as an excitable system, such as Morris-Lecar, Hodgkin–Huxley, Hindmarsh–Rose or Fitzhugh-Nagumo, being the last one the model used. The ordinary differential equations that define the FHN model are the following: ˙y=−y(y−1)(y−a) + w+u ˙w=αy +βw (8.1) Where yand ware the state variables and uthe input variable also known as forced term. The u variable is later referred as Iapp, and it represents the inhibition and excitation relations between the specific neuron and its neighbours. Solving the system using Euler’s method, for a given step size h, the resulting equations are: yn+1 =yn+h(−yn(yn−1)(yn−a) + wn+Iapp) wn+1 =wn+ (αyn+βwn)(8.2) In order to train the system appropriately, the input Iapp will remain constant enough time so that the system reaches a stable point or a stable orbit, therefore the system has enough time to reach these two stable configurations and will not waste time in an already explored state, that is, the Iapp changes to explore the new state as soon as the previous one is totally stable. 8.1 Three inputs case To be able to predict the FHN model, the WN has to get an initial condition composed by any yand wand then output the next yand wpoint in the same Iapp input, as if the system was solved using Euler’s method. With an initial ynand wn, it will calculate the yn+1 and wn+1 of the same Iapp. Then the WN will be able to predict any yn+1 and wn+1 from any given Iapp with just one initial condition. This idea is represented with the WN’s structure shown in figure 8-1, seen below. Figure 8-1. WN’s structure for the FHN model. page 22 Manuscript To achieve this, the WN has to be trained with random Iapps and one period delay outputs, as inputs, to be able to calculate the weights’ vector. The reason why it is necessary to randomise the Iapps used to train the WN is because it has to learn from different transient states, to comprise all possible responses from how the system is externally excited. And the reason why the two other inputs are a one period delay outputs is a way to stick to the Euler’s method nature, for solving the system and generating data to create a target the WN can predict. To approximate the FHN model through an entire grid of points in every Iapp from any possible initial grid’s point, the WN has to explore a considerable range for the variables y,wand Iapp, visualising it as a 3D space. To do it so, there are two possible ways. The first one consists in defining every single point of the 3D space with the coordinates w,yand Iapp as inputs and then calculate the next integration step of every one of them using Euler and using the result as a target for the output. This first method is computationally intense, on that account a second way is thought. This second one consists in one single initial condition used to solve a given number of integration steps from the Euler’s method in a single Iapp and then change to the next Iapp using the last yand wresult as the new initial condition in the following Iapp and repeating the process a given number of different Iapps. Finally, the wavelets used in the WN are the hat, the quadratic spline and the bicubic spline, seen in table 4-1. Thehat is simpleandcan be used forfast results butthe C1class functionthat defines itcan make it hard for a first prediction if the WN does not make a good solution for the weights’ vector. Whereas the quadratic and the bicubic splines have a non-null second derivative and they have smoother solutions to every next resolution level. As the FHN model is non-linear, it might be a good option to work with spline wavelets. The quadratic should be enough for a good prediction but for the sake of fun, the bicubic is also proposed to see if it can have other benefits over the quadratic spline. 8.2 Activation function superposition Since now, the WN’s growth is expressed as a result of a single activation function defined in the domain of the scale function used. Using a single activation function makes the WN less capable to approximate a function with high accuracy rather than using the same activation function superposed a number of times in the same domain, due to the fact that the extremes of de domain the scale function takes values near zero and these values define the wavelet matrix. As it is studied in [10], there are a series of advantages by using superposed functions, such as the need for less resolution levels for an accurate result and an accurate approximated target using less sampled data. In this project it is studied the impact of the number of superposed scale functions using a hat, a quadratic spline and a bicubic spline. In figure 8-2 it can be seen the approach used for the superposition using a quadratic spline as an example. There is always a single function that exists through all the domain and then the same function is added in symmetrical translations. A Prediction Model for Neuronal Synaptic Inputs page 23 Figure 8-2. Superposed quadratic spline scale functions. Looking at figure 8-2, it can be understood that a weight multiplied by a small value in the wavelet matrix F(x)will have a small impact compared to the weights associated to values near the centre of the domain, obtaining worse results for the approximated target function. As wavelets are expressed as a sum of scale functions, as seen in (5.4), the superposition property is also applied in wavelets by just superposing the scale functions. In the second level of resolution and further the activation function will be a superposition of wavelets. With this final approach the WN’s growth has to be redefined, using the expression in (6.11), as shown in the equation (8.3). ∆Wavelons0=JE(1 + ∆Wavelons)(8.3) Where: J: is the number of superposed functions. E: is the number of the WN’s inputs. ∆Wavelons: is the wavelons’ increment, already defined in the chapter 6.4. ∆Wavelons0: is the new wavelons’ increment. page 30 Manuscript 9 Economic balance In this chapter it is computed the costs associated to the development of the project taking into account human resources, the engineering such as modelling and development of the WN, any licence if used and the electricity bill. The number of hours is charged with a 50€/h price. Simulation is also considered as important as any other engineering part due to the fact that while simulations are running it cannot be changed and tested anything about the current model. In this project no licence is used, as the software needed it is been Ubuntu 19.10, a free Linux distribution, and emacs as a text editor for Python 3.7 coding. The report writing has been done in LaTeX, a free text editor widely used for papers and many types of publications and no paid Python library was needed. Finally, the electricity bill is added to the total cost, not only counting the hours spent by human resources and engineering but also considering the hours spent with the various simulations done with the WN. The computed cost is shown in table 9.1, with a total cost of 20755,73€. Table 9.1. Economic balance of the project. The consumption considered for the electricity bill is computed with the consumption under average load of the computer used during the project [12] and the mean price per kWh in a year in Spain [13]. A Prediction Model for Neuronal Synaptic Inputs page 31 10 Environmental Impact The environmental indicator that reflects the impact of the greenhouse gases emitted directly or indirectly, corresponds to the carbon footprint, which is measured in the equivalent CO2mass. To know the range of the footprint it is going to be taken in consideration the indirect emission due to the consumed electricity to fulfil the project. Following the methodology described in PAS 2050:2011 [14] by the British Standard Institution (BSI) Group, who define the specifications to evaluate the greenhouse gases emissions, it is used the factor of 0,385 kgCO2eq/kWh. Using the kWh defined in table 9.1 to indicate the consumption during the project and the factor provided by the BSI, it can be found the total mass of CO2emitted in the atmosphere during the realization of this project, shown in table 10.1. Table 10.1. Carbon footprint of the project. page 32 Manuscript 11 Conclusions To sum up, the final results show that it is been possible to create a working WN based on the FHN model, it has been able to predict the behaviour of a black box excitable system, like one described by a neuron, following the studies conducted in [3]. All the objectives have been achieved. As it has been exposed, the concepts about the theory of wavelets have been understood deeply enough to use them to define an algorithm useful for the WN’s purpose. Developing the empirical growth rate for the wavelons, based on the number of inputs, has given a simplistic point of view for writing the algorithm’s code around it. Many WN’s structures have been studied to be able to know which one is suited for a desired scenario, as shown in chapter 8.3. The error-time graphic can help to decide how much error can be sacrificed in favour of a reduction in time, useful for future studies and development computationally intensive. Studying the implementation of a linear term, as seen in chapter 8.4, has also been positive for achieving a slightly reduction in training time and a significant reduction in error, for non-linear scale functions’ implementation in the WN. Notwithstanding the three months spent in this project, it has been nearly impossible try to implement a parallelized work flow to increase the computational performance, nonetheless is was improved the by using the minimum number of variables and preallocated variables to reduce the work done by the memory. As a final thought for future work, it can be a good idea to explore the use of a second neural network to work beside the WN. The second NN could be responsible for finding the best frequency for the Iapp to make the system reach, as fast as possible, to stable point with less oscillations, whereas for an Iapp in a stable orbit could have a fixed frequency. And there will be no need to fix a small frequency, enough to explore every possible system response as it is already done. A Prediction Model for Neuronal Synaptic Inputs page 33 12 References [1] T. P. Trappenberg, “Fundamentals of Computational Neuroscience,” 2010. [2] M. S. Keil, E. Roca-Moreno, and Á. Rodriguez-Vázquez, “A neural model of the locust visual system for detection of object approaches with real-world scenes,” Proceedings of the Fourth IASTED International Conference on Visualization, Imaging, and Image Processing, no. February, pp. 340–345, 2004. [3] N. Roqueiro, C. Claumann, A. Guillamon, and E. Fossas, “A Black-box Model for Neurons,” 2019 IEEE 10th Latin American Symposium on Circuits and Systems, LASCAS 2019 - Proceedings, pp. 129–132, 2019. [4] M. Bergman, “Containing the sun,” Harvard, p. 1, 2019. [Online]. Available: https://news.harvard.edu/gazette/story/2019/04/ harvard-princeton-scientists-make-ai-breakthrough-for-fusion-energy/ [5] A. Tch, “The mostly complete chart of Neural Networks, explained,” 2017. [Online]. Available: https://towardsdatascience.com/ the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464 [6] S. Verma, “Understanding different Loss Functions for Neural Networks,” 2019. [Online]. Available: https://towardsdatascience.com/ understanding-different-loss-functions-for-neural-networks-dd1ed0274718 [7] G. Mallat, “A Theory for Multiresolution Signal Decomposition : The Wavelet Representation,” vol. II, no. 7, 1989. [8] A. D. Señales, “Introducción a la Transformada Wavelet,” ... de Señales y sistemas. Universidad de Navarra, 2006. [Online]. Available: http://www.exa.unicen.edu.ar/ escuelapav/cursos/wavelets/apunte.pdf [9] S. T. Ali, J.-P. Antoine, and J.-P. Gazeau, “Multidimensional Wavelets,” pp. 307–330, 2000. [10] C. A. Claumann, “Desenvolvimento E Aplicações De Redes Neurais Wavelets E Da Teoria De Regularização Na Modelagem,” p. 167, 2003. [11] A. K. Alexandridis and A. D. Zapranis, “Wavelet Neural Networks : A Practical Guide,” 2006. [12] F. Glaser, “Análisis completo del Asus ROG Strix GL702ZC (Ryzen 7 1700, Radeon RX 580),” p. 1, 2017. [Online]. Available: https://www.notebookcheck.org/ Analisis-completo-del-Asus-ROG-Strix-GL702ZC-Ryzen-7-1700-Radeon-RX-580.248850. 0.html [13] Tarifasgasluz, “¿Cuánto cuesta la luz al mes?” p. 1, 2020. [Online]. Available: https://tarifasgasluz.com/faq/cuanto-cuesta-luz-mes{#}:{~}:text= Elpreciomediomensualdel,propiosdescuentos{%}2Cpromocionesycondiciones. [14] BSI, “PAS 2050:2011 Specification for the assessment of the life cycle greenhouse gas emissions of goods and services. British Standards Institution, London,” pp. 1–45, 2011.