Learning how to divert attention in multilingual machine translation to mitigate gender bias

Bachelor’s thesis - Engineering Physics ETSETB - Universitat Politècnica de Catalunya Learning how to Divert Attention in Multilingual Machine Translation to Mitigate Gender Bias Author: Pablo Crucera Barrero Project supervisors: Marta Ruiz Costa-jussà and Carlos Escolano Peinado June of 2021 Abstract Deep learning has aroused over the past years as a promising technique in AI. Implementing algorithms that allow systems to learn with applications in several fields has made a difference with the conventional rule coding that had multiple limitations. Since successful experiments within this scope have taken place, the limits of the development of intelligent machines seem to have faded out. However, this is still not a flawless procedure: there is a great number of improvements ahead to be performed in order to obtain better results. One of these issues is the presence of stereotypes in large data sets, a generalized problem in many applications of Deep Learning. In the case of Neural Machine Translation, this phenomenon leads to gender-biased inaccurate translations. In this thesis, our commitment is to work on developing a modification of the attention mechanism (within the architecture of NMT) that can mitigate such bias. Attention controls the amount of context that is used in Neural Machine Translation. The motivation of this work is that the attention mechanism for different types of words, e.g. words with different amounts of dependencies, should be different. The concentration or diversion of context information both at encoding and decoding levels may differ depending on the studied biases and language. We study the effect of controlling this amount of attention both by modifying the attention mechanism at the encoding and decoding level, for self and inter-attention. We evaluate this for the case of mitigating biases in a multilingual setting on a recent NMT architecture: the Transformer. The results show a similar quality of translation as the baseline model in terms of accuracy, but outperforms the reference model in nearly all gender-bias indicators, yielding results that are encouraging for continuing exploring this line of research. 2 Contents Abstract 2 Contents 3 Acknowledgements 5 1.Introduction 6 1.1.Contributions 6 1.2.Thesis organization 7 2.Theoretical framework 8 2.1.Fundamentals of Deep Learning 8 2.2.Neural machine translation (NMT) 9 2.3.Review of the state-of-the-art 10 2.3.1.CNN-based architectures 10 2.3.1.1.ByteNet 10 2.3.1.2.Convolutional Seq2Seq 10 2.3.2.An approach to attention 11 2.3.3.The Transformer 12 2.3.3.1.Overall model architecture 12 2.3.3.2.Multi-head attention 14 2.3.3.3.Scaled Dot-Product Attention 14 3.Methodology 19 3.1.How to modify attention? An adaptive temperature parameter 20 3.2.Experimental implementation 23 4.Experiments 25 4.1.Datasets and Preprocessing 25 4.1.1.BLEU score 25 4.1.2.WinoMT 25 4.2.Baseline model 26 4.2.1.Model settings 26 4.2.2.Results 26 3 4.3.First experiment 29 4.3.1.Approach 29 4.3.2.Results and conclusions 29 4.4.Second experiment 31 4.4.1.Approach 31 4.4.2.Results and conclusions 31 4.4.3.Translation examples 34 5.Conclusions, further research and personal evaluation 36 Bibliography/References 37 4 Acknowledgements First of all, I would like to thank my project supervisors: Marta Ruiz Costa-jussà for bringing me the opportunity of being part of her research group and getting into a world that I was looking forward to exploring (Deep Learning) despite knowing about my scarce background in the field, as well as enriching my project with her vast experience doing research; and Carlos Escolano Peinado, for offering me a helping hand when I needed it the most and being always there for bringing me assessment on programming (which requires not only knowledge in computer science but also tons of patience), apart from providing me with numerous ideas when some experiments failed. I also would like to thank Christine Basta, a member of the research group, for helping me as well in confusing moments regardless of not being one of the project supervisors. My colleagues (and friends) Graciela Ojeda and Júlia Sánchez have also made contributions in this thesis, in the form of moral support and sharing common ideas. Finally, I would like to thank my parents’ task: although they had no idea of the theory behind this project and they struggle with English, they have been willing to read this report and have always reminded me that I can go as far as I want. 5 1.Introduction We are living sweet times for the development of AI. Within the past decade, there have taken place several successful experiments within the scope of AI and, specially, of Deep Learning. The growth that this field is experiencing and the fact that there are being implemented endless applications of the techniques used in it makes us think that Artificial Intelligence has come to stay. However, there is a lot yet to be explored. One the one hand, there are applications ahead to be discovered in which AI could play an interesting role. On the other hand, despite the improvements in the state-of-the-art techniques, there is still an interesting work to be done for boosting up some imperfect procedures that concern branches of AI such as Machine Learning (and Deep Learning). Among the list of non-flawless applications of AI there is Machine Translation (MT), a subfield of Natural Language Processing (NLP), which is specially being developed thanks to the breakthrough of Neural Machine Translation (NMT). There is a constant appearance of new NMT architectures trying to dethrone previous models in terms of translation accuracy, computational cost of the translation or model training time, which is a proof of the existence of numerous possible improvements that could take place within this scope. 1.1.Contributions One of the imperfections that we have found is the presence of biases in NMT models, and particularly, gender bias. Most models are trained with large datasets composed of millions of sentences in which the male gender is sometimes employed as the neutral gender (default) and which mostly preserve gender stereotypes. This phenomenon causes translations not to be neutral when it comes to gender. In this work, we will try to tackle this problem by implementing modifications in a NMT architecture called The Transformer[1], which relies entirely on the concept of attention. In particular, we are due to modify the attention mechanism (both self-attention and cross-attention) that this model makes use of at the encoding and decoding levels in a multilingual setting. We believe that this will not only help to mitigate biases but also will improve the quality of the translation. 6 1.2.Thesis organization This work is organized according to the order of appearance of the necessary concepts to understand the following sections. First of all, we will review the basic concepts behind the field of our approach, as well as we will review the state-of-the-art models and techniques that we will employ in the experiments. After that, we will present the proposed modifications in the attention mechanism, how they work and how we will implement them. Finally we will cover the baseline model and the results obtained both in this reference model and our experimental implementations, both in terms of translation accuracy and gender bias results. To get a better understanding of what is to be implemented, we must first introduce a set of theory fundamentals that are described in the next section: 7 2.Theoretical framework 2.1.Fundamentals of Deep Learning Machine learning is a sub-field of AI which has as a goal to provide computing systems with the ability to perform certain tasks (classification, regression, etc.) through learning. In this process, the machine is fed with examples that will help it to improve in its task. Within this field, we can find Deep learning, a set of algorithms which aims to correctly model tasks that require a high degree of abstraction. The basic unit of what this algorithm uses is the perceptron. The perceptron is a mathematical modeling of a biological neuron that maps a set of inputs (x1, x2,…, xN) to an output yin the following way: (1) Where w1,w2,…,wNare a set of weights, w0is what we refer to as bias and fis our activation function (up to our choice, but some activation functions like tanh on ReLU are commonly used). Deep learning uses cascading layers of perceptrons, building a complex structure that is called Neural Network. In between each layer, there is what we call a hidden state. Figure 2.1: an example of a simple neural network with three layers, four different inputs, one output and a hidden state consisting of five elements. Circumferences represent perceptrons, while the arrows represent the connectivity between them. 8 Neural networks produce outputs that will fit our task better or not depending on the values of the weights that are assigned. In order to optimize the values of the weights, we must define a loss function that will be the mathematical criterion for the machine to optimize the weights. The algorithm used to modify the weights while ensuring the decrease in the loss function is typically gradient descent. Since there are infinite configurations of neural network structures, several types of neural networks have been described. In the ‘Review of the state-of-the-art’ section, we will mention some of them related to a concept that is of our interest: Neural Machine Translation. 2.2.Neural machine translation (NMT) The field that we are working on is Natural Language Processing (NLP). NLP refers to the branch of computer science, and more specifically, to the branch of artificial intelligence (AI), which deals with giving computers the ability to understand spoken text and words in the same way as human beings. NLP combines computational linguistics (rule-based modeling of human language) with statistical, machine learning, and deep learning models.[2] Within this field we can find Neural Machine Translation (NMT), which is the machine translation task (generating automatic translations given a source text in a certain language to another language) that makes use of artificial neural networks for trying to predict what the best possible output is. Several NMT architectures have been described, most of them in the very few past years. We are due to explain some of the cutting-edge architectures that outperformed their predecessors. 9 Softmax After that, a softmax function is applied. Softmax is a function commonly employed in neural networks, specially in the last steps, that is used for mapping a set of input values to a normalized set of outputs of the same size. This can be seen as the creation of a probability distribution from an original set of compatibility scores whose order of magnitude is arbitrary1. The formula for this function is defined as follows: (3) The numerator is strictly positive and increasing with xigiven the nature of the exponential function and the denominator is a constant value greater or equal than the numerator. This means that regardless of the sign of the input, each of the components of the output will be within a bounded domain between 0 and 1, and it is trivial to see that the sum of all components of the output yields 1. In addition, the bigger a component is (compared to the others), the more likely it becomes. In the previous step of this process we committed to explain the importance of the scaling factor . The main reason why this division is performed relies on the exponential behavior of the softmax function. The authors of this attention function argue that the input sequences (queries and keys) can be modelled as random signals of mean 0 and variance 1, and therefore, a scalar product between two dk-sized vectors would yield a random output of mean 0 and variance dk.[1] For large values of dk(i.e. long sentences), the variance growth would cause some input components to the softmax to tend to get significantly larger than others. This phenomenon would be more pronounced after the application of the softmax regarding the exponential nature of this function. The effect of a softmax on a random input signal as the one mentioned before with and without the scaling factor can be observed in Figure 2.4: 1In our case, the order of magnitude will not be arbitrary but will be according to a probability distribution with parameters that will be described later on 16 Figure 2.4: simulation of the output of a softmax function applied over a set of random compatibility scores for a word and a sequence of length N=20, with and without the effect of the scaling factor In Figure 2.4 we simulate the effect of the softmax upon a set of compatibility of scores of a word with the words in its context, which correspond to the first subplot. The output probabilities of the softmax function without including the scaling factor dk(classical dot-product attention) are depicted in the subplot in the middle, while the last subplot shows how this factor affects the output probabilities of the softmax. It can be observed that after applying the scaling the attention acquires a less steep profile, which means that attention has been diverted. We place so much emphasis on this because our methodology will take advantage of this phenomenon, as it will be explained in further detail in 3.1. In the case of the masked multi-head attention, which is only used in the decoder, masking is performed before the softmax function. Remember that the reason for doing this is to prevent from attending to subsequent positions of the output (words that have not been determined yet). This is done by filling the compatibility scores of elements of the output corresponding to future time steps with sentinel values tending to -∞, so the attention score becomes 0 after the softmax. 17 Matrix multiplication Finally, the output of the softmax function is multiplied by the matrix Vusing another simple matrix multiplication. Therefore, the Scaled Dot-Product Attention module in fact produces a weighted sum of value projections of word embeddings, in which weights are the probabilities for these values to be the next element of the output sequence. 18 3.Methodology Regarding the main purpose of this thesis, which is reducing gender bias for machine translation, we would like to obtain the desired accurate translations for sentences in which there could be stereotypes. The presence of such bias is mostly given for certain words in the translation from languages that do not specify the gender of the person/animal in the word itself but in the context to languages that use a different word for each gender. For instance, the English word nurse could be translated into Spanish as enfermero (masculine word for nurse) or enfermera (feminine noun). However, this translation, if generated automatically, will tend to produce a femenine noun since this profession is traditionally associated with the feminine gender, sometimes ignoring the hints in the context suggesting that this person might be a boy. Regarding that attention refers to the amount of context used for generating this translation, we believe that using the appropriate attention, the model would be able to detect the gender of the subject by focusing on key words. Let us see another example of what a diverted or focused attention is, this time relating attention with gender. For example, the sentence “The doctor asked the nurse if he could help her.” contains two pronouns that prove that the doctor is a woman and the nurse is a man. However, the traditional gender roles for these professions are the other way around. A focused attention would produce an output that tends to be more similar to a word-by-word translation, and for the words ‘doctor’ and ‘nurse’ in particular, the model would tend to translate them as the most likely output for those single words, which correspond to the targets that have been repeated the most to the model in the training process. Since we are working with enormous datasets which preserve gender stereotypes, a focused attention mechanism will produce, for instance, the following sentence for English-Spanish translation: “El doctor pidió a la enfermera si él podía ayudarla.” In this example, the model has not related the nouns to the pronouns in the context, yielding a misinterpretation of the genders of both professionals. However, a more diverted attention would take into account these pronouns for the translation of the nouns, which would help us obtain a more accurate translation like: “La doctora pidió al enfermero si podía ayudarla.” Nevertheless, an excess of diversion is not something positive either: over-relating words that have nothing to do with each other would produce wrong outputs with completely senseless meanings. Therefore, we are due to modify the attention mechanism in an intelligent way in order to produce better results. 19 3.1.How to modify attention? An adaptive temperature parameter In the previous section, we mathematically described attention as a probability score that is computed for every word according to how much it does relate to the others. From now on, we will call the output of the last step of the computation of the function (softmax) attention weights. Our main idea stands that including a temperature parameter for the attention weights could regulate the amount of context that should be used for producing the output given a certain input. Recalling the formula for Scaled Dot-Product Attention, the effect of this parameter will be implemented as follows: (4) Where 𝜏, the temperature parameter, is calculated as: (5) Being fsome activation function and wij an element of a set of learned weights. A different temperature factor is computed for each sentence in the input sequence. The justification for this relies on the idea that such a parameter could help the model redistribute the attention in a similar way that the scaling factor applied before the softmax does. As shown in Figure 3.1, a higher temperature will produce smaller gradients than those scaled by , leading to smoother attention distributions. Similarly, a low temperature will 1 𝑑𝑘 generate steep peaks. 20 Figure 3.1: simulation of the effect of including a temperature parameter on the attention weights for a random set of words and different temperatures around 1 (raw attention weights). Words are represented along the x-axis, temperature changes along the y-axis and the attention weights are shown in the vertical direction. Figure 3.1 shows the attention weights (z-axis) relating a certain word with the N=20 words in its context (represented along the x-axis) for different temperatures, which vary along the y-axis. What is of particular interest from this graph is to compare how attention changes comparing it to the special case of 𝜏=1 (the temperature effect does not modify the attention weights). As it can be observed, the word(s) that were most relevant for the translation of the word that we are referring to become even more important when temperature decreases (attention becomes more focused). However, as the temperature increases, the attention becomes more planar, which means that it is diverted. Despite improving the results for long sequences, we believe that this single value might 1 𝑑𝑘 not be the optimal scaling factor at all times: our hypothesis states that for some cases it will be better to either concentrate or divert attention, and our control mechanism to do it will be 𝜏. Since we are still unable to predict what the optimal value of 𝜏is, we will try to compute it using artificial intelligence: taking as an input , a layer of a neural network will produce 21 a calculation of a temperature parameter that will be optimized using a typical Deep Learning algorithm, gradient descent. 22 3.2.Experimental implementation Every input sentence is represented by a matrix: every row would represent the embedding of a specific word. In practice, the input text of the model is a tensor of order three, for which a new dimension is added so it contains the information of a different sentence in each of the components[1]. ●Each component of the first dimension represents a different sentence. Therefore, its size is called Batch size. ●The second dimension is related to the number of words per sentence. Since it is more practical to use cubic data structures, all sentences must have the same length. To counteract this obstacle, a padding effect is used in order to leave blank spaces in phrases, letting us deal with sentences of different sizes. ●Finally, the third dimension is the word embedding, the size of which is h·dk=h·dv=dmodel . Recall that we are working with eight different heads, each of which uses a dv-sized vector to represent a word. These heads are packed together in the same matrix concatenated in a single dimension. Because we want a temperature parameter for each of the sentences, the output that we want to produce before dividing by our factor is in fact a vector of length Batch size. Then, each matrix representing a sentence (each element in the first dimension is a matrix that stores the embedding of a sentence or its projection) must produce a scalar parameter. How can this be implemented? By now, our assumption is that this temperature factor can be calculated as a weighted linear sum of each element in the matrix, and after that, set to an activation function. These weights would be learned for optimal results, so in fact, we could see these operations as the effect of a single layer of a neural network applied to a vector that results from the concatenation of rows or layers. One of the drawbacks in this procedure is the fact that the size of the layer is computed as the product between the size of the second dimension (length of sentences) and the size of the third one (length of the word embedding for each head), but it must be kept constant as well. Since these two dimensions depend on non-fixed parameters, we should find a way to turn a matrix of variable dimensions to a fixed-size input matrix. A way to do this is using an Adaptive Pooling filter in 2D, which is an effective method that uses discretization by selecting values among a set of components. The values that are produced could be selected by either selecting max values or performing an average calculation on them. The disadvantage of doing this is that, after the pooling filter, some information will be copied in case of shorter inputs than the output matrix or missing in case of longer sequences. The overall procedure that we will implement is depicted in Figure 3.2: 23 Figure 3.2: schematic sketch showing the main steps of our implementation In Figure 3.2 we intend to represent the steps described above. Each colored grid of the input represents the attention weights at a certain time step for a sentence. From these matrices we obtain square matrices with fixed size (represented by the 5x5 grid stack) by pooling them. Then, a linear layer computes a weighted sum of each of the elements, which is then set to an activation function. The output of this function (denoted by f) will be our temperature parameter. 24 4.Experiments 4.1.Datasets and Preprocessing For our experiments we will use the Europarl v7 corpus[11] as the training set of sentences, a dataset containing millions of preprocessed examples of sentence translation between the main European languages. For the validation and testing we will use newtest2012 and newtest20132datasets. All data will be preprocessed using standard Moses[12] scripts. More concretely, the languages of our choice will be English, French, German and Spanish. We will outline the results of the task for all language pairs (both at source and target languages) using the BLEU score as our criteria for the quality of the translation. In addition, we will employ the WinoMT[13] algorithm for detecting biases, using WinoBias[14] and Winogender[15] as the datasets for these criteria. 4.1.1.BLEU score Bilingual evaluation understudy, commonly known as BLEU, is a method employed for measuring the quality of a translation from a certain source language to another.[16] This algorithm computes the number of correctly predicted outputs given a target sequence and a candidate sequence, the accuracy of which we want to measure. In our case, we will employ BLEU-4, which takes into account the precision of the translation of each single word, word pairs and 3-grams and 4-grams of words, as well as introduces a penalty factor for short sentences. 4.1.2.WinoMT WinoMT[13] is an algorithm developed in order to measure gender biases taking as an input the translation of three different text datasets written in English: a set of 3888 sentences with equal presence of masculine and femenine genders and some neutral nouns (from now on, neutral), a set of pro-stereotypically biased sentences (from now on, pro) and the anti-stereotypically version of the same set (from now on, anti). It consists of the evaluation of three parameters: ●Accuracy: measures the percentage of genders predicted correctly from the neutral dataset in the target language. ●ΔG: computes the difference between the F1scores (that is, a balanced parameter that takes into account both precision and recall) of male and femenine nouns. ●ΔS: computes the difference between the accuracies of pro and anti. 2Available at http://www.statmt.org/ 25 However, things change when the temperature is only applied in the encoder. In this case, the best results have been obtained for the configuration which combined the averaging pooling filter with the bigger layer (sized 529). After 14 epochs, the valid loss becomes 59.6279, which is a value that represents a slight increase of 0.51% with respect to the best loss in the baseline model. We believe that this value would have outperformed the baseline loss if it had been trained for as many epochs.3 The BLEU4 scores obtained for all language pairs are presented in Table 4.3. Target language EN ES FR DE Average EN - 29.48 29.31 21.72 26.84 ES 27.39 - 29.89 19.99 25.76 FR 26.14 29.06 - 19.26 24.82 DE 23.95 24.86 25.23 - 24.68 Average 25.83 27.80 28.14 20.32 25.52 Table 4.3: BLEU4 score obtained in our second experiment, using 𝜆=4, an averaging pooling filter and a layer size of 529 A graphical comparison between the baseline and this model results is shown in Figure 4.2. 3The training process did not reach as many iterations as in 4.2.2 due to technical issues of the machine. 32 Figure 4.2: comparison between the BLEU4 scores obtained both in the baseline and this experiment As one can observe from Table 4.3 and Figure 4.2, our model shows a slightly lower performance than the baseline model, yielding an average BLEU4 0.35 below than our reference value. However, this difference is not much significant regarding the fact that the two models have been trained for different times.4 Let us now see the results of the WinoMT scores for this model in Table 4.4. Accuracy ΔG ΔS ES 53.8% 15.7 6.2 FR 43.8% 17.5 16.4 DE 57.4% 1.1 0.5 Average 51.7% 11.4 7.7 Table 4.4: WinoMT score obtained in our baseline model. Accuracy denotes the percentage of correctly predicted genders in the translation, ΔG denotes the difference between masuline and femenine F1scores and ΔS denotes the difference between the pro-stereotypical and anti-stereotypical gender assignments. The underlined results are those whose values outperform the baseline model. 4In addition, the BLEU4 score is subject to the target sentence used as a reference in the algorithm and provides underestimated results for candidate sentences which contain the same meaning as the source but differ in form from the target. We will see later that this phenomenon has occured when testing this model. 33 The comparison between these values and those obtained in 4.2.2 are shown in Table 4.5: Baseline Experimental model Difference (%) Accuracy (Avg.) 52.6% 51.7% -1.7% (relative to 52.6) ΔG (Avg.) 15.4 11.4 -26.0% ΔS (Avg.) 9.8 7.7 -21.4% Table 4.5: comparison between the average WinoMT scores obtained in our baseline model and the experimental model. We observe that our model does mostly outperform the baseline in terms of gender. The values of ΔG and ΔS, which are our gender bias indicators, have decreased in average with respect to the baseline for all languages (except for ΔG for French). However, gender prediction accuracy is on average slightly lower than the baseline model. The conclusion that we draw from these facts is that our model has not been able to outperform the baseline in the overall quality of translation, but the modifications that have been implemented in the attention mechanism have somehow contributed to the debiasing of the original model. Our prototype does not pay as much attention to the stereotypes in the training dataset as the reference model, which is an indicator that we have fulfilled our initial commitment for this work. As an additional note, it would have been pleasing to have had the model tested after being trained for as long as the baseline. 4.4.3.Translation examples In this section we attach significant en-es and es-en translations that would help us prove that the model works correctly despite the BLEU4 score, which is not a perfect algorithm since using synonyms is penalized regardless of the fact that it might be a successful translation. 34 Source sentence Target sentence Hypothesis desde este último punto de vista , todas las tecnologías actuales no son lo suficientemente buenas para nosotros ; esto da más trabajo a los diseñadores de algoritmos . from the latter point of view , all current technologies are not good enough for us - this adds work to the algorithm designers . from this last point of view , all the current technologies are not good enough for us ; this gives more work to algorithms designers . Norway has handled its oil wealth very carefully - all but a small percentage of money from the industry is invested in a special fund for the benefit of future generations . Noruega ha gestionado su riqueza petrolífera con mucha cautela ; todo , excepto un pequeño porcentaje del dinero procedente de esta industria , se invierte en un fondo especial en beneficio de las futuras generaciones . Noruega ha gestionado muy cuidadosamente su riqueza petrolera : todo menos un pequeño porcentaje de dinero del sector se invierte en un fondo especial en beneficio de las generaciones futuras . the summit also concluded with the joint commitment of Chile and Peru to accept a ruling by the Hague Court to adjudicate a border dispute between the two countries la cumbre concluyó también con el compromiso conjunto de Chile y Perú de aceptar un fallo de la Corte de La Haya que dirima un diferendo fronterizo entre ambos países . la Cumbre concluyó también con el compromiso común de Chile y Perú de aceptar una sentencia del Tribunal de La Haya para zanjar una disputa fronteriza entre los dos países . las teorías liberales y los medios de comunicación afirman hasta la saciedad que el estado no debería aportar capital a la economía , que dirigir la economía conduce a su fracaso . Liberal theory and the Media incessantly claim that the State may not participate with capital in its own economy , and that a controlled economy leads to economic ruin liberal theories and the media claim ad nauseam that the state should not bring capital to the economy , that leading the economy leads to its failure . Table 4.6: translation examples between English and Spanish 35 5.Conclusions, further research and personal evaluation The first thing that we have been able to appreciate not in the experiments but in the review of the state of the art is the fact that attention is a powerful mechanism. In fact, it is so solid that some cutting-edge NMT models rely entirely on it. However, imprecisions also arise in the task of translation using these architectures, being gender bias one of these imperfections. Throughout this work, we have presented possible modifications in the attention mechanism used in The Transformer that could help the model neutralize this inaccuracy using Deep Learning techniques in a multilingual setting. The most significant results are obtained when modifying the attention in the decoder, for the case of employing averaging pooling filters and big layer sizes. In our best setup, despite a slight drop in the BLEU score with respect to the original model, a decrease in the gender bias indicators takes place, which is a symptom that the approach that we have presented is in the right direction. More concretely, we have managed to decrease the average ΔG score by 26% and the average ΔS score by 21.4% for German, French and Spanish. It is important to point out that the training process of this latter model did not go as far as the baseline which we have compared our results with. As a future research, it would be interesting to see how this model performs after carrying out the whole training. Other intriguing possible modifications could be trying other values of 𝜆, which has remained constant in all executions for our second experiment, or finding out what the main reason for the quality and loss drops in the implementations at the decoder is. This has been a rewarding project regardless of the results, which I consider to be good despite the difficulties. I had never entered the world of Machine Learning (and even less of Deep Learning) before, nor in the field of NLP, and therefore the world of NMT has been totally new to me. Also, I had never worked with Python until I started this project. The lack of knowledge of all of these things have made me spend considerable time catching up with the necessary background for the project. The fact that the remote machine in which we were doing the experiments collapsed several times during the training process (which takes days to complete) did not help either. That is why I consider that it has not been an easy job for me and that the learning process that I have experienced has been intense. But, as I have said, I have learnt so many things that were new to me that have made from this experience a worthwhile one. I look forward to continuing learning more about AI and, in particular, the applications of Deep Learning so I acquire sufficient knowledge for carrying out ambitious research like this project on my own. 36 Bibliography/References [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, Attention is all you need. arXiv:1706.03762, 2017. [2] IBM, Natural Language Processing. [Consultation date: 02-06-2021, available at https://www.ibm.com/cloud/learn/natural-language-processing] [3] W. Zhang, K. Itoh, J. Tanida and Y. Ichioka, Parallel distributed processing model with local space-invariant interconnections and its optical architecture. Appl. Opt. 29, 4790-4797, 1990. [4] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. van den Oord, A. Graves and K. Kavukcuoglu, Neural Machine Translation in Linear Time. arXiv:1610.10099, 2016. [5] J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. N. Dauphin, Convolutional Sequence to Sequence Learning. arXiv:1705.03122, 2017. [6] H. Lamba, Intuitive Understanding of Attention Mechanism in Deep Learning. [Consultation date: 17-04-2021, available at https://towardsdatascience.com/intuitive-understanding-of-attention-mechanism-in-deep-lear ning-6c9482aecf4f]. [7] D. Bahdanau, K. Cho and Y. Bengio, Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473, 2014. [8] T. Luong, H. Pham and C. D. Manning, Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015. [9] L. Weng, Attention? Attention! [Consultation date: 19-04-2021, available at https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html]. [10] J. Alamar, The Illustrated Transformer. [Consultation date: 22-04-2021, available at https://jalammar.github.io/illustrated-transformer]. [11] P. Koehn, Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86, 2005. [12] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens et al. Moses: Open source toolkit for statistical machine translation. In Proceedings of the ACL: Demo Papers, pages 177–180, 2017. [13] G. Stanovsky, N. A. Smith and L. Zettlemoyer, Evaluating Gender Bias in Machine Translation. arXiv:1906.00591, 2019. [14] R. Rudinger, J. Naradowsky, B. Leonard and B. Van Durme, Gender Bias in Coreference Resolution. In Proceedings of the ACL: Short Papers, pages 8-14, 2018. 37 [15] J. Zhao, T. Wang, M. Yatskar, V. Ordoñez, K. W. Chang, Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the ACL: Short Papers, pages 15-20, 2018. [16] K. Papineni, S. Roukos, T. Ward and W. J. Zhu, BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311-318, 2002. [17] C. Escolano, M. R. Costa-jussà, J. A. R. Fonollosa, M. Artetxe, Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders. arXiv:2004.06575, 2020. [18] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier and M. Auli, fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019. [19] R. Müller, S. Kornblith and G. E. Hinton, When does label smoothing help? Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. [20] D. P. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014. 38