Intelligent data aggregation using autoencoders and other statistics

Title: Advanced Methods for Optical Constellation Analysis and Compression Author: Raúl López Martínez Advisor: Luis Velasco and Marc Ruiz Department: Computer Architecture University: UPC Academic year: 2021-2022 Interuniversity Master in Statistics and Operations Research UPC-UB Advanced Methods for Optical Constellation Analysis and Compression palabra 2 Universitat Polit`ecnica de Catalunya Facultat de Matem`atiques i Estad´ıstica Master’s Degree in Statistics and Operations Research Master’s Degree Thesis Advanced Methods for Optical Constellation Analysis and Compression Ra´ul L´opez Mart´ınez Supervised by Luis Velasco and Marc Ruiz September, 2022 Thanks to all the teachers and students of the faculty, they were one of the keys reasons for me to love these studies. Thanks to my project advisors for the interest they put on this project. Thanks to Basetis for encouraging me to keep studying and extending my knowledge. Thanks to my family for always supporting my decisions during these last 2 years and for educating me in the values that made me the person who I am today. Thanks to all my friends and beloved ones, it is always a pleasure be surrounded by such nice people. Special thanks to Aitor Munar and Ti`a Roig for being incredible human beings. Thanks for all the moments we shared and everything I learnt from them. We miss you. Rest in peace. Abstract Optical constellations offer a highly dimensional representation of the signals in optical transport network technologies and they can be analyzed for several use cases such as optical network health analysis and secure optical networks. It is crucial for operators to implement efficient monitoring architectures that mitigate potential drawbacks (such as high capacity exhaustion) while ensuring the validity and reliability of widely collected monitoring data. In this project we aim to provide robust techniques for optical constellation analysis and compression achieving large compression rates with negligible information loss. In particular, optical constellations will be characterised through parametric probability distributions and compressed through autoencoder architectures. Keywords optical networks, optical constellation, machine learning, deep learning, neural networks, dimensionality reduction, autoencoders 1 palabra 2 Contents 1 Introduction 6 1.1 Motivation........................................... 6 1.2 Previouswork ......................................... 6 1.3 Objectives ........................................... 6 1.4 Documentorganisation .................................... 7 2 Background 8 2.1 OpticalConstellationdata................................... 8 2.2 Mixturedistributions ..................................... 11 2.3 MachineLearning ....................................... 12 2.4 DimensionalityReduction................................... 14 2.5 Summary............................................ 16 3 Preprocess 17 3.1 The need to preprocess the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Spatialdistributions...................................... 17 3.3 Results............................................. 20 3.4 Summary............................................ 22 4 Dimensionality reduction: data reconstruction 24 4.1 GaussianMixtures....................................... 24 4.2 Imagedata........................................... 25 4.3 Results............................................. 26 4.4 Summary............................................ 31 5 Use case: Data augmentation 32 5.1 Motivation........................................... 32 5.2 Objectives ........................................... 32 5.3 Approach............................................ 33 5.4 Results............................................. 33 5.5 Summary............................................ 36 6 Concluding remarks 37 6.1 Conclusions .......................................... 37 6.2 Personalthoughts....................................... 37 6.3 Futurework .......................................... 38 3 Advanced Methods for Optical Constellation Analysis and Compression Figure 2: Sequence of spans in a single link scenario (source: [7]). The dataset distinguishes between three configurations of the simulator: optimal, sub-optimal and degradation. The different configurations introduce power variations that result in small changes in the optical constellations without worsening lightpaths’ QoT. The dataset consists of a set of optical constellation instances, each of them conformed by a set of 2048 points in the complex plane. There are 1250 instances from optimal and 500 from each of sub-optimal and degradation scenarios. Figure 3: One of the constellation diagrams in OCATA 16QAM-singleLink-s1 dataset (source: [7]). For the execution of this project, the data has been splitted (in a stratified manner) so that 20% is kept for testing the performance of the developed models. 10 2.2 Mixture distributions In probability theory, a mixture distribution[8] is the probability distribution of a random variable which definition depends on a collection of other random variables in the following way: •First, a random variable from the collection is selected according to given probabilities of selection. •Finally, the value of the randomly selected random variable is realized. The underlying random variables can be univariate or multivariate (random vectors), having all of them the same dimension. In the second case, the mixture distribution would be a multivariate distribution. Given a finite or countable set I, probability density functions {fi( x)}i∈Iand weights {πi}i∈Isuch that πi≥0 and Pi∈Iπi= 1, the mixture distribution is defined by its probability density function f( x) = X i∈I πi·fi( x) Usually, the mixture components are not arbitrary probability distributions but members of a parametric family (for example gaussians). In this case the density function can be written in a sum form as: f( x; θ) = X i∈I πi·fi( x; θi) where  θiis the vector of parameters that defines the distribution of the component iof the mixture and  θ is the concatenation of all the parameters { θi}i∈I. Although we have only covered the case in which the collection of component distributions is countable, there’s an analogous kind of distributions for the case in which it is uncountable: compound probability distribution. However, in this project we will be working with the particular case of finite mixtures where components are from the same parametric family. At this point, one can already notice that mixtures of 16 2D-gaussian distributions can be a good fit to characterize the samples of 16QAM optical constellation diagrams (like the ones in the OCATA 6QAMsingleLink-s1 dataset) since their symbols should be located in clusters around the theorical position of the 16 constellation points. Figure 4: Density of a 2D Gaussian Mixture with three components. 11 Advanced Methods for Optical Constellation Analysis and Compression 2.3 Machine Learning Machine Learning (ML) is a field that falls in the intersection between probability theory, statistics, optimization and computer science. It is devoted to understanding and building methods that learn from data, i.e, methods that use data to improve the performance of a mathematical model on particular tasks [9]. Depending on the nature of the tasks, we can distinguish between different types of machine learning algorithms. The two main groups are supervised and unsupervised algorithms. •Supervised Learning: We say that a machine learning algorithm is supervised if it is trained through input-output example pairs. •Unsupervised Learning: Unsupervised learning algorithms are the ones that that learn patterns from untagged data. The machine builds latent representations of data through mimicry and then it is able to generate imaginative content from it. The main machine learning algorithms that will be used in this project fall into the field of unsupervised learning. •Other: Other supervision levels in the spectrum are semi-supervised learning and reinforcement learning. Semi-supervised learning comes from the impossibility to have all your data tagged while still wanting to extract the value from the untagged data. It combines unsupervised and supervised algorithms. Reinforcement Learning (RL) is concerned with how intelligent agents might act in an environment in order to maximize a notion of cumulative reward. The cumulative reward is nothing else than a performance score that the agent receives from the environment as feedback depending on the outcomes of the taken actions. Generalized linear models In statistics, a generalized linear model (GLM) is a generalization of ordinary linear regression. The GLM generalizes linear regression by making the linear model to be related to the response variable through a link function and by allowing the magnitude of the variance of the measurements to be a function of the predicted values. The particular GLM case of ordinary linear regression takes place when the link function is the identity. GLMs main advantages are that they are easy to implement, fast to fit and highly explainable. In a GLM, we assume that the outcome Yof the dependent variables Xis generated from a particular distribution in an exponential family. The mean of the response distribution depends on the independent variables through the equation: E(Y|X) = µ=g−1(X·β) where E(Y|X) is the expected value of Yconditional on X,X·βis the linear predictor and gis the link function. GMLs require that the link function is monotonic and differentiable over the range of possible values of µ. The unknown parameters of GLM, i.e the vector β, are meant to be estimated through data. 12 In this project, linear models will be used to get baseline performance metrics that will be aimed to be beaten by more complex models explained next. Deep Learning The term Deep Learning refers to a family of machine learning methods based on Artificial Neural Networks with representation learning. Learning can be supervised, semi-supervised or unsupervised. Artificial Neural Networks are composed by multiple layers, which are finite collections of GLMs also known as neurons, to progressively extract higher-level features from the raw input. The Deep Learning approach is currently state-of-the-art in pretty much every of the machine learning and AI problems that people do research in. Nevertheless, it usually needs a vast amount of data and computational resources to achieve such performance. Figure 5: Graphical schema of an Artificial Neural Network This is an example of a particular type of Artificial Neural Networks where there are no cycles in the edges that join neurons, what is called a feed-forward network. The other main type of ANN, recurrent neural networks, have recurrent connections between neurons and they are used in problems where time dependencies of the input have to be taken into account by the network. To easily understand how Neural Networks work, it is better to focus on what goes on in a neuron of a hidden or output layer. Figure 6: The computations that take place in a neuron 13 Advanced Methods for Optical Constellation Analysis and Compression This could be an artificial neural network that is inside the previous one. The output of a neuron is a GLM of its input values, with its weights and link function. In deep learning the term link function is not used, instead, they call activation function (sigma in the figure above) to its inverse. A very common (but not the only) activation function is the sigmoid, which leads to a neuron: a=σ(Pωixi) where σ(z) = 1 1+e−x. The weights is what an algorithm aims to learn. The weights of the whole networks are initialised (randomly or not) and they are fine-tuned to optimise the performance of the network in a specific task based on some training data. In this project, deep learning will be the go to approach to try to beat the achievements of (generalized) linear model baselines. 2.4 Dimensionality Reduction The term Dimensionality Reduction refers to mathematical transformations of data that transform it from a high-dimensional space into a lower-dimensional one, also known as latent space, so that the latent representation retains the important properties of the original data. Ideally, the dimension of the latent space should be close to the intrinsic dimension of the data. Working in high-dimensional spaces can be undesirable for many reasons: •As a consequence of the curse of dimensionality, raw data is often sparse. •Analyzing such data can even be computationally intractable. Dimensionality reduction application shines in fields that deal with large numbers of variables and/or observations. Some of these fields are signal processing, speech recognition, neuroinformatics, and bioinformatics. In this project, dimensionality reduction techniques will be applied to optical constellations data so the project falls into the field of signal processing. Among all the possibilities that Machine Learning offer in terms of reduction of dimension models, we will be using Principal Component Analysis (PCA) as a baseline linear model and Deep Autoencoders will be used to try to beat PCAs performance. Principal Components Analysis The Principal Components of a finite set of points that live in a real coordinate space are a sequence of p unit vectors. They are defined so that the i-th vector is the direction of the line that best fits the data while being orthogonal to the first i−1 vectors. The best-fitting lines are obtained by least squares regression, therefore they are the ones that minimize the average squared euclidean distance from the points to the line. The directions through this iterative method constitute an orthonormal basis of the p-dimensional space where the raw data lives in. Principal component analysis (PCA) stands for the process of computing the principal components of some data and using them to perform a change of basis on the data. 14 PCA is commonly used for exploratory data analysis purposes and for dimensionality reduction by projecting the data points onto only the first ˆp<pprincipal components. This way, lower-dimensional data is obtained while preserving as much of the data’s variance as possible. This method has the advantage that fast to train, easy to implement and results are interpretable. However, its main drawback is that it assumes that the components are orthogonal linear combinations of the original features and this assumption is not held in some kinds of data. Figure 7: Illustration of the obtained Principal Components on 2-dimensional data. Deep Autoencoders An encoder is any function that maps some data into a lower-dimensional space. A decoder is a function that does the same but from the lower-dimensional space to the original input space. Any concatenation of an encoder and a decoder is known as an autoencoder architecture. When the encoder and decoder are neural networks, we say that the autoencoder is a deep autoencoder. A Deep Autoencoder is a type of deep learning model that is used to learn efficient encodings of unlabeled data (unsupervised learning). The encoding is refined during the train phase by attempting to regenerate the original input from the generated encodings (or codes) through the decoder. This way, the network learns lower-dimensional representations of the data by ignoring the insignificant noise of the data instances. Although in this project we will be using autoencoders mainly for dimensionality reduction purposes, they are also applied to many other problems, including facial recognition, feature detection, anomaly detection and synthetic data generation. 15 Advanced Methods for Optical Constellation Analysis and Compression Figure 8: Schema of an autoencoder network. 2.5 Summary Optical constellations is a way in which signals can be represented. In this project, we will be working with a dataset of optical constellations: the OCATA 16QAM-singleLink-s1 dataset. This dataset is public and contains samples of simulated optical constellations under different simulator configurations. When one wants to characterise optical constellation samples by the spatial probability distribution of its symbols mixture distributions can be proved to be a good fit. The problem that we will attempt to solve on this data through machine learning is dimensionality reduction, which falls into the field of unsupervised learning. To do so, the different machine learning models that will be trained are PCA and Deep Autoencoders. 16 3. Preprocess In this chapter, first, it is going to be discussed why it is necessary to preprocess the optical constellations before applying most of dimensionality reduction models. Then, one preprocess approach will be suggested to be used in the incoming chapters. Finally, from the suggested approach, different families of models will be compared according to some reasonable metric. 3.1 The need to preprocess the data The samples of the dataset come in tabular format with 2048 columns that contain the values of the symbols. The column in which a symbol is located is random since the signals are demodulated also in random order. Therefore, the columns of the dataset lack of meaning and the samples are just an unordered set of demodulated symbols. Machine learning models can only learn from meaningful features so, if it is desired to train machine learning models with it, it is needed to do some sort of data engineering in order to characterize the samples through another set of features. This feature engineering step will be referenced to as preprocess in this document. 3.2 Spatial distributions One can think of a sample as a sample of size 2048 of a random variable that follows a parametric distribution with values in R2. The parameters of the distributions can be estimated through the symbols of the sample and they can be used as a set of features. Two families of spatial distributions will be proposed, both of them being mixture distributions: •Gaussian Mixtures As we have seen in the background chapter, the 2D points of the samples are supposed to to conform 16 clusters, each of them centered at one of the 16 constellation points of the 16QAM optical constellation. That is why it makes sense to fit 2D Gaussian Mixtures with 16 components. It is done through the Expectation Maximization algorithm and the means of the gaussians are initialised at their theorical location: the location of the constellation points of the 16QAM optical constellation. This way, we obtain a mixture f(x0,x1) = 15 X k=0 πkfNk(x0,x1) where fNk( x)=2πDet(Σk)−1 2exp(−1 2( x−µk)TΣ−1 k( x−µk)) is the density function of the component kof the Gaussian Mixture. 17 Advanced Methods for Optical Constellation Analysis and Compression Figure 9: Optical constellation diagram (left) and a visualization of its corresponding Gaussian Mixture features (right). •Mixtures of uniform distributions The most natural way to visualize the samples as humans is in a 2D-scatterplot. By doing that, we get rid of the lack of order in the features of the dataset. This way to get rid of the lack of order for visualizing the data can be used also for preprocessing the data. Besides, this is a more general approach than Gaussian Mixtures since there is no assumption that the demodulation noise that makes a symbol not to be located in the theorical constellation point position is gaussian. Taking into account that there has been a lot of research in computer vision models and that a sample can well represented by an image it makes sense to preprocess the samples into images. The way that we will do so is by computing the 2D histogram of the samples with a determinate number of bins (the number of bins determines the resolution of the image). Hence, a preprocessed sample would be a n binsx×n binsyimage where the value of each pixel is the number of points of the sample that fall into that point of the discretization of the space. We divide all the values by 2048 (the total number of points) so that all the values sum up to one and they define a discrete probability distribution where each value is the probability that a sampled point falls in the corresponding 2D bin of the histogram. We obtain a discrete probability distribution p(x∈bink) = πk where the MLE of {πk}kis their empirical value ˆπk=nk 2048 the proportion of the sampled points that fall into the corresponding bin. 18 Figure 10: Optical constellation diagram (left) and its corresponding 2D Histogram (right). At this point, when we already have a discrete distribution, we need to turn it into continuous since that the nature of the theorical random variable. We can do so by assuming an uniform distribution over the space that encloses a bin of the 2D histogram. That leads to the mixture fπ(x0,x1) = X k∈{bins} πk 1(x0,x1)∈k Ak =πk∗ Ak∗ where Akis the area of the bin kand k∗is the bin in which (x0,x1) falls into. We will be using squared bins of the same size so that they have all the same area. Therefore fπ(x0,x1) = 1 AX k∈{bins} πk1(x0,x1)∈k=πk∗ A where Ais the area that every bin has. Since the only parameters are {πk}k, it can be easily derived that MLE in this case is also ˆπk=nk 2048 Note that here we are proposing one possible discretization of the 2D space to be applied equally to all the data samples. Instead, one could define an adaptive way to discretize the data that takes into account particularities of an input to decide in which way it is going to be discretized. Anyways, the selected approach can be as precise as one wants if the discretization is refined enough. That is proved by the following limit computation: lim A→0fπ(x0,x1) = lim A→0 πk∗ A= +∞ since πk∗is bounded between 0 and 1 (and value 0 can be excluded). Therefore, the limit of the likelihood of a model given a sample is also infinite: lim A→0L(π| X) = lim A→0Y (x0,x1)∈ X fπ(x0,x1) = Y (x0,x1)∈ X lim A→0fπ(x0,x1)=+∞ 19 Advanced Methods for Optical Constellation Analysis and Compression where targetkrefers the value of the original mixture of uniforms in bin k(analogous for predk) and Ais the (constant) area of a bin. Note that, for every bin, the first term of the product is 2048 ·targetk=nkthe number of points in bin kin the original sample. The second term is the log density of the reconstructed distribution in bin kand it is divided by 2048 to apply the same re-scaling as we did in the preprocess chapter. The 2048 factors cancel one with each other and the last formula is obtained. One can expect this autoencoder model to perform better than the ones previously introduced for two main reasons: is trained in the same metric in which it is going to be evaluated and the output does never need any post-process to be transformed in a valid distribution. 4.3 Results All the previously mentioned dimensionality reduction models have been trained setting the number of dimensions of the latent space (bottleneck size) to different values. For more details on the hyperparameters of the models check the model hyperparameters section of the only appendix in this document. The performance of the compressors will be compared in terms of their goodness-of-fit, taking the original Gaussian Mixtures as the baseline models. Evaluation The main goodness-of-fit metric will be the log likelihood of the reconstructed spatial distributions. The size of the latent space is also going to be something to take into account by looking at the how much we reduce the dimension of the feature space with respect to the Gaussian Mixture baseline. This will be measured through the ratio of the number of parameters of Gaussian Mixtures divided by the bottleneck size. This ratio will be referred to as compressing factor. Results Let’s first analyse the performance of the PCA autoencoder on the Gaussian Mixture features. Figure 13: Likelihood of the PCA reconstruction of Gaussian Mixtures. Figure 13 displays the performance of the PCA autoencoder on the Gaussian Mixture features (reconsX stands for the reconstructed distributions with bottleneck size X). Note that the reconstruction of the 26 small distance samples is harder for the model in this case. It has to do with the low variance gaussians being closer to be degenerate distributions [10]. Note also how the PCA is able to get good reconstructions when increasing the latent space number of dimensions. PCA on gaussian features doesn’t seem to be the go to approach since one can expect to get better compression rates on small distance connection constellations. Next, let’s discuss the results of the PCA autoencoders on the image data. Figure 14: Likelihood of PCA reconstruction of image data. In this case, as seen in Figure 14, PCA reconstructions of image data are generally less precise than the original Gaussian Mixtures. It is also remarkable that increasing the number of dimensions in the latent space leads to worse reconstructions. It is not contradictory since, as it has been commented previously, PCA optimizes the MSE of the parameter reconstructions instead of the log likelihood of the reconstructed distributions. This approach gives reconstructed distributions that are clearly worse than the baseline Gaussian Mixtures so it won’t be considered further in the project. Moving into the analysis of the performance of Deep Autoencoders on 64x64 images, as seen in figure 15 and comparing the shown results with the previously discussed ones, the reconstructed distributions are more robust in the sense that the expected decay of the log likelihood with the increase of the total distance in the samples is observed on both train and test set. Besides, the reconstructions are as good as the original Gaussian Mixtures or better than them (in terms of log likelihood) for total distance values higher than 700 also on both sets. This behaviour is observed for all the tried values for the number of dimensions of the latent space. 27 Advanced Methods for Optical Constellation Analysis and Compression Figure 15: Likelihood of Deep Autoencoders’ reconstructions of 64x64 images. Figure 16: Likelihood of Deep Autoencoders’ reconstructions of 128x128 images. Next, figure 16 illustrates the obtained results for Deep Autoencoders trained on 128 ×128 images where a similar behaviour is observed for all the tried bottleneck sizes. The deep autoencoder approach is also a robust dimensionality reduction method for 128 ×128 images according to the log likelihood of the reconstructed distributions. Furthermore, the reconstructions are better than the original Gaussian Mixtures for distances larger than approximately 500. Moreover, on the samples with total distance smaller than 500, the performance is pretty similar to the one of the original Gaussian Mixtures. We see how the log likelihood gap that we had on the previous figure for those samples has been closed through increasing the resolution of the images. Figure 17 shows the compressing factor of the evolution of the mean log likelihood trained models computed on the test set as we increase the compressing factor. This figure clearly shows that the Deep Learning approach (denoted by the term DL in the figure legend) is the one that gives best results, leading to reconstructions that are on average better than the original gaussian mixtures with high compressing factor. At this point of the project, we can just keep the deep autoencoders with bottleneck size 3 for 64x64 and 128x128 images which lead to a compressing factor of 31.67 with respect to the original Gaussian Mixtures and a factor 1365.33 with respect to the raw optical constellation. The evolution of the log likelihood with respect to the connection total distance are displayed in table 2 and figure 18. 28 Figure 17: Log likelihood of the selected Deep Autoencoders. Distance interval Gaussian Mixture 64x64 Images 128x128 Images 64x64 recons 128x128 recons 0-500 -1.192 -1.253 -0.915 -1.385 -1.271 501-1000 -2.348 -2.149 -1.651 -2.362 -2.312 1001-1500 -2.855 -2.520 -1.871 -2.824 -2.802 1501-2000 -3.212 -2.758 -1.988 -3.149 -3.117 Table 2: Mean log likelihood of the reconstructed distributions through the selected autoencoders. Figure 18: Log likelihood of the selected Deep Autoencoders. 29 Advanced Methods for Optical Constellation Analysis and Compression The proposed compression models achieve performance metrics that are comparable to the ones of the baseline Gaussian Mixtures, being them better for large distance samples and slightly worse for small distance ones. As the image resolution is increased from 64 ×64 to 128 ×128, the log likelihood of the reconstructed small distance images gets really close to the Gaussian Mixtures’ ones. Figure 19: Three examples of original images and their autoencoded images. It is also interesting to have a look on the visualization of the reconstructed distributions. Figure 19 shows three examples of original images and reconstructed distributions for both of the used image resolutions. There are various remarkable aspects in this figure: •Autoencoders are trained to detect patterns in the optical constellations and remove what could be random noise in the reconstructions. Note how the reconstructed distributions concentrate most of the density in the 16 centers of the theorical constellation points. This also happens when one (heuristically) assumes a Gaussian Mixture model. •For large distance connections, we are more likely to obtain reconstructed distributions that differ more from a Gaussian Mixture pattern, which might not always be a valid assumption. Through image data, we are using a less-restrictive model in the sense that the dispersion around the theorical constellation points may be not gaussian-like. This is one of reason why this kind of modelling leads to better results for large distance connections. •Small distance samples reconstructions look pretty similar for both of the image resolutions. Nevertheless, we know that the 128 ×128 are generally better. That comes from the fact that the 30 discretization done in 128 ×128 images allows for more precision and the obtained densities through reconstruction can fit better the actual constellations. Conclusions To sum things up, the conclusions that one can take from the obtained results are the following: •The approach that seems to provide a more robust solution to the data reconstruction problem is the use of Deep Autoencoders on image data with a custom loss function that makes the autoencoder to train directly to train on the log likelihood of the reconstructions. •The Deep Autoencoders’ performance on 128x128 image data is similar to the original Gaussian Mixtures’. It is remarkable that such performance is obtained with a compressing factor of 31.67 on the number of parameters with respect to the Gaussian Mixtures’ ones. •On 64x64 images, Deep Autoencoders give similar but slightly worse results, being this difference larger the smaller the total distance of the connection. 4.4 Summary When aiming to use techniques to reduce the dimension data one has to choose how to preprocess the data that will feed the machine learning models. Gaussian mixtures’ parameters have constraints that suppose a problem when one wants to autoencode them since a generic decoder won’t always generate sets of parameters where the constraints are fulfilled. Nevertheless, the constraints on the parameters of the mixture of uniforms are easier to deal with. Principal Component Analysis (with a post-processing of the output in order to fulfill constraints) has been evaluated on data reconstruction via log likelihood of the reconstructed distributions and it doesn’t seem to perform pretty well neither in gaussian nor in uniform mixtures. One has to take into account that PCA optimizes MSE of the reconstructed parameters instead of log likelihood of the reconstructed distribution. On the other hand, the deep learning approach gives more robust reconstructions of the original distributions. The fact that deep autoencoders are directly trained to maximize the log likelihood of the reconstructed distributions via a custom loss function is key to obtaining such good performance. Moreover, the application of non-linear models (unlike PCA) also helps to boost the performance. This approach let us get reconstructed distributions that are comparable (or slightly better) to the baseline Gaussian while using more than 31 times less features. 31 Advanced Methods for Optical Constellation Analysis and Compression 5. Use case: Data augmentation This chapter aims to present, in a proof of concept manner, an use case of autoencoders on optical constellation data. In particular, autoencoders will be used for data augmentation [11]. First, the motivation and objectives of this use case will be introduced. Next, the technical approach will be presented. Finally, the performance of the models will be evaluated and some conclusions will be drawn from the results. 5.1 Motivation The computation behind the generation of the optical constellation simulated data requires a large amount of computational resources, which scales with the sampling rate of the signal processing. In such scenario, it is reasonable to aim to extract all the potential value from the data. One thing one can try to do is to use the simulated data to generate synthetic data. The practise of extending a dataset with synthetic data is also known as Data Augmentation and that can lead to boosts in the performance of other ML-based applications. Data augmentation is a technique to artificially create new training data from existing training data. This is done by applying domain-specific techniques to examples from the training data that create new and different training examples. In machine learning, generally, the higher the dimension of the feature space, the more training examples are needed. In cases of high dimension feature space, which is the case of our image data, data augmentation is usually applied in order to have more compact training sets in both the latent and the original input space. 5.2 Objectives Ideally one would like to one would like to be able to reconstruct the what a sample with 2048 symbols would look like from another with, let’s say, only 1024 symbols. Second ones are simulated way more efficiently. Instead, it has been decided to solve a different problem: reconstructing the spatial distributions (preprocessed samples) using samples that only have half of the symbols symbols (1024). This would allow to extend the original dataset with two kinds of synthetic data: •Encoded random subsets of 1024 symbols of the original samples. •Encoded simulated samples that only contain 1024 symbols. The objective is to develop autoencoders that are able to generate synthetic spatial distributions in order to enlarge the size of the original train set. 32 5.3 Approach Taking into account that the Deep Autoencoders that where developed in the previous chapter showed good performance it makes sense to use the same architecture. That is going to be the approach in this chapter. The fact that we will be dealing with a neural network architecture (which allows on-line training [12]) gives us the following three different training scenarios: •Zero-shot pretrained models: use the pretrained models from the previous chapter without any additional fine-tuning for this particular task. •Fine-tuned pretrained models: load the pretrained models from the previous chapter and fine-tune them for this particular task. •Trained from zero models: initialise the weights of the models randomly and train them only on this particular task All these training scenarios will be evaluated on the same test set. 5.4 Results All the previously mentioned approaches have been trained for 64 ×64 and 128 ×128 images. The latent space number of dimensions will be set to 8 for 64 ×64 images and 13 for 128 ×128 images since the last chapter’s results suggest it is a good option. Their performance is going to be evaluated according to different criteria. Evaluation The test set for this task consists of, for each optical constellation of the original test set, the sample plus a set of 50 optical constellations with only 1024 symbols that are random subsets of the symbols in the original sample. The performance of the Data Augmentation autoencoders will be evaluated on the log likelihood of the reconstructed spatial distributions from the samples with only 1024 symbols given the original test optical constellation. In this case, having likely enough reconstructions would mean that we generally only need half of the symbols to infer the spatial distribution of the complete optical constellation. Results Figure 20 compares the three training scenarios that were introduced previously for the fitted autoencoders. Notice how the zero-shot learning autoencoder performs pretty well but the other ones have slightly higher log likelihood among the test set examples. It is decided to just keep the pretrained and fine-tuned autoencoders. 33 Advanced Methods for Optical Constellation Analysis and Compression Figure 20: Log likelihood of autoencoders for data augmentation on different training scenarios. Figure 21 displays the log likelihood of the reconstructed masked images with respect to the original distributions. The main conclusion one can take from these results is that both image resolutions lead to reconstructed distribution of the masked samples that are approximately as likely as the original Gaussian Mixture models that were fitted to the complete samples. It is also remarkable that, when working with 128x128 images, the results are slightly better for samples that come from low-distance connections. Figure 21: Log likelihood of the data augmentation autoencoders compared to the original spatial distributions’. The respective numeric results are shown in table 3, where one can also notice that the obtained log likelihood values are also similar to the ones of the reconstructed distributions from the non-masked images. Indeed, the reconstructed distributions from the masked images can even be equal to the reconstructed distributions from the original images to a regular human eye and figure 22 illustrates so. 34 Distance Gaussian Mixt. 64x64 recons 128x128 recons masked 64x64 recons masked 128x128 recons 0-500 -1.192 -1.385 -1.271 -1.383 -1.281 501-1000 -2.348 -2.362 -2.312 -2.355 -2.330 1001-1500 -2.855 -2.824 -2.802 -2.821 -2.813 1501-2000 -3.212 -3.149 -3.117 -3.143 -3.144 Table 3: Log likelihood of the reconstructed masked images through the selected autoencoders, compared with the non-masked reconstructions. Figure 22: Three example images and reconstructions with their respective masked images and reconstructions. Conclusions After having a look at all the obtained results, these are the main conclusions that can be drawn from them: •It is not necessary to simulate 2048 symbols samples in order to obtain likely spatial distributions of the optical constellations. The proposed autoencoder-based approach generates spatial distributions through samples that have only half of the symbols and the obtained results on the test set show that they are comparable to the original Gaussian Mixtures in terms of likelihood of the probabilistic models. •Once again, now being in the masked optical constellations scenario, higher resolution images 35