Online Learning Techniques for Neural Translation Systems

Escola Tècnica Superior d’Enginyeria Informàtica Universitat Politècnica de València Online Learning Techniques for Neural Translation Systems DEGREE FINAL WORK Degree in Computer Engineering Author: Kevin Montalvá Minguet Tutor: Francisco Casacuberta Nolla Álvaro Peris Abril Course 2015-2016 Resum Els sistemes de traducció automàtica s’han fet servir desde la seva concepció per traductors professionals per a accelerar i facilitar la seva tasca. Aquestos sistemes reben traduccions editades professionalment a través del seu ús, lo que pot pontecialment millorar el seu rendiment. En les últimes décades, Xarxes Neuronals Artificials s’han utilitzat per a desenvolupar sistemes de traducció automàtica complets. En aquest treball s’han utilitzat dos algoritmes d’aprenentatge online per a millorar traductors neuronals, que ja havien acabat l’etapa d’entrenament. Una amplia gamma d’experiments s’han dut a terme per trobar els hiperparàmetres óptims per els algoritmes en cada tasca de traducció, i després el rendiment d’aquestos sistemes, adaptats amb cadascun dels algoritmes aprenentatge (configurats amb aquestos hiperparàmetres óptims), s’ha mesurat abans i després en quatre tasques de traducció, i s’han extret conclusions sobre com han millorat o empitjorat. S’ha modificat el codi del traductor neuronal, s’ha implementat un dels algoritmes presentants i s’ha desenvolupat un nou codi per oferir una interfície per a fer experiments de forma automatitzada. Una altra familia d’algoritmes ha sigut implementada i probada amb els models i tasques disponibles, sense resultats positius. Finalment, línies futures d’investigació en adaptació de traductors neuronals han sigut considerades i discutides al final d’aquest treball, a més de la seva situació en l’estat actual de la traducció automàtica. Paraules clau: aprenentatge automàtic, xarxes neuronals, xarxes neuronals recurrents, traducció automàtica, traducció neuronal Resumen Los sistemas de traducción automática han sido utilizados desde su concepción por traductores profesionales para acelerar y facilitar su tarea. Estos sistemas reciben traducciones editadas profesionalmente a través de su uso, que pueden potencialmente mejorar su rendimiento. En las últimas décadas, se han utilizado redes neuronales artificiales para desarrollar sistemas de traducción automática completos con éxito. En este trabajo se han usado dos algoritmos de aprendizaje online para mejorar redes neuronales ya entrenadas. Un amplio abanico de experimentos se ha llevado a cabo para encontrar el conjunto óptimo de hiperpárametros para los algoritmos en cada tarea, y se ha calculado el rendimiento de estos sistemas, adaptados con cada algoritmo con sus hiperparámetros óptimos encontrados de forma empírica. Se han extraído conclusiones acerca de la mejora o empeoramiento de los traductores. Se ha llevado a cabo una modificación de la base de código del traductor neuronal, junto con la implementación de uno de los algoritmos y el desarrollo de una nueva herramienta para proporcionar una interfaz para realizer experimentos de forma automatizada. Otra familia de algoritmos ha sido implementada y probada con los modelos y tareas disponibles, con resultados insatisfactorios. Finalmente, líneas potenciales de investigación en adaptación de traductores neuronales han sido consideradas y discutidas al final de este trabajo, junto con su situación en el estado actual de la traducción automática. Palabras clave: aprendizaje automático, redes neuronales, redes neuronales recurrentes, traducción automática, traducción neuronal iii iv Abstract Machine Translation systems have been used since their inception by professional translators to speed up and ease their work. Those systems receive professionally edited translations through their use, which could potentially improve their performance. In the last decades, Artificial Neural Networks have been used to develop complete Machine Translation systems to great success. In this work two online learning algorithms for Artificial Neural Networks have been used to enhance already trained neural translators. A wide array of experiments have been carried out to find the optimal hyperparameters for the algorithms in each task, and then the performance of those systems, adapted with each algorithm with their empirically found optimal set of hyperparameters, has been measured before and after in four translation tasks, and conclusions have been extracted on how they improved or worsened. A modification of a neural translator codebase has been carried out, along with the implementation of one of the algorithms and the development of new codebase to provide an interface to perform experiments in an automated way. One additional family of algorithms has been implemented and tested with the available model and tasks to no avail. Finally, possible future lines of research on adaptation of neural translators have been considered and discussed at the end of this work, along with their situation in the current state of Machine Translation. Key words: machine learning, neural networks, recurrent neural networks, machine translation, neural translation Contents Contents v List of Figures vii List of Tables vii List of Algorithms viii 1 Introduction 1 1.1 Background .................................... 1 1.2 Motivation..................................... 1 1.3 Goals ........................................ 2 1.4 Structure...................................... 3 2 State of the art 5 2.1 Statistical Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Neural Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.1 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.2 Bidirectional Recurrent Neural Network . . . . . . . . . . . . . . . 10 2.2.3 Encoder-Decoder model . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.4 Training .................................. 12 3 Online learning 13 3.1 Training Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Online learning algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.1 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 AdaGrad.................................. 14 3.2.3 Passive-Aggressive............................ 14 4 Experiments 19 4.1 Software ...................................... 19 4.1.1 Theano................................... 19 4.1.2 GroundHog................................ 19 4.2 Experimentation framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3 Hyperparametersearch ............................. 23 4.3.1 Xeroxtask................................. 23 4.3.2 EUtask................................... 25 4.4 Testsetexperiments................................ 27 5 Conclusions and future work 29 Appendices Acronyms 31 A. Acronyms ..................................... 31 Bibliography 33 v List of Figures 2.1 Example of Recurrent Neural Network. . . . . . . . . . . . . . . . . . . . . 7 2.2 Visualization of a RNN unfolded in time. . . . . . . . . . . . . . . . . . . . 8 2.3 IllustrationoftheGRU............................... 9 2.4 StructureofaBRNN................................ 10 4.1 Differences in BLEU in test set. . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2 Execution time of experiments in the test sets. . . . . . . . . . . . . . . . . . 28 List of Tables 4.1 Corpora characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.2 InitialBLEU..................................... 23 4.3 Hyperparameter search for the Xerox task, from English to Spanish, with SGDalgorithm. .................................. 24 4.4 Hyperparameter search for the Xerox task, from Spanish to English, with SGDalgorithm. .................................. 24 4.5 Hyperparameter search for the Xerox task, from English to Spanish, with AdaGradalgorithm. ............................... 24 4.6 Hyperparameter search for the Xerox task, from Spanish to English, with AdaGradalgorithm. ............................... 25 4.7 Hyperparameter search for the EU task, from English to Spanish, with SGD algorithm. ..................................... 25 4.8 Hyperparameter search for the EU task, from Spanish to English, with SGD algorithm. ..................................... 26 4.9 Hyperparameter search for the EU task, from English to Spanish, with AdaGradalgorithm. ............................... 26 4.10 Hyperparameter search for the EU task, from Spanish to English, with AdaGradalgorithm. ............................... 26 4.11 BLEU score obtained by retraining on the test set................ 27 vii List of Algorithms 3.1 Passive-Aggressive approximation pseudocode. . . . . . . . . . . . . . . . 17 4.1 Passive-Aggressive approximation execution flow. . . . . . . . . . . . . . . 21 viii CHAPTER 1 Introduction 1.1 Background In 1949 Warren Weaver laid the foundations for Machine Translation (MT) by proposing the use of computers to tackle the challenge of translation. Consequently, in the 50s and 60s, several attempts were made to create practical translators, and by the mid sixties the newly created ALPAC (Automatic Language Processing Advisory Committee)[15] published a report in which they painted the future of MT as bleak. About 10 years later nonetheless, after the conception of some new approaches and the increase in processing power of computers, there was a renewal of the expectancies for MT, and research and interest rose back up. So far, MT is a challenge that has been engaged with multiple approaches and methodologies, but most of them can, nowadays, be classified into three categories[10]: RuleBased approaches, Corpus-Based ones and a hybrid approach in between. Rule-Based Systems (RBS) require the use of human knowledge on languages, and the use of grammars and vocabularies. Corpus-Based Systems (CBSs) use big parallel corpora of text for the system to learn them and reproduce their features. A big advantage of CBSs over Rule-Based Systems (RBSs) is that language data is readily available for almost any language in the world, a big amount of information in several languages. From books to news, movie scripts, trial transcripts and laws, there is a large number of sources from which data can be extracted that Corpus-Based translators can use to improve the quality of their product. Since CBSs are trained on data, they also have (usually) the possibility of scaling up and out with hardware, while RBSs are usually limited by the quality of their rules. 1.2 Motivation Professional translators use MT systems regularly to speed up their task. The translations obtained through those systems though are of irregular quality, and many if not most of them must be amended by human translators to obtain a satisfactory result. Those corrections can be used to improve the MT system being used to, in the long term, reduce the number of corrections that the professional needs to apply to future translations. The limit being of course “perfect translations”, where no corrections are needed. The correction of machine-generated translations human translators is known as postedition. In a post-edition context, the human translator inputs a source sentence, the software generates a hypothesis in the target language and the human corrects the translation. In a simple system this correction is only useful to the user, since the system will 1 8State of the art x1 h1 y1 x2 h2 y2 xt ht−1 yt ... ht Input layer Hidden layer Output layer Figure 2.2: Visualization of a RNN unfolded in time. xtstands for the t-th input processed by the network (alternatively, at the t-th iteration), htis the hidden state of the network at iteration t and ytis the output of the network at iteration t. ht=fh(xt,ht−1)(2.3) yt=fo(ht)(2.4) fhand fodepend on the architecture of the network and the choice of activation functions. RNNs, however, have one well-known shortcoming when it comes to sequence processing: the vanishing gradient problem[5]. Gradient-based training algorithms update the network weights in proportion to the gradient of the error function with respect to each of them. Since layers “further away” from the output layer (in terms of connections) have smaller contributions to output values, this aforementioned gradient is in turn smaller. In the case of sequence-processing RNNs, this means that information from elements far “in the past” is almost lost. To solve this problem, many RNN-based systems have successfully used the RNN architectures that are about to be discussed. In the late 90s, a family of RNNs appeared, called Long Short Term Memory (LSTM) networks [14], which have shown very good results in many sequence learning tasks. They make use of a set of gates to keep some kind of memory. LSTM units are trained to retain different degrees of memory, which allows RNNs to successfully remember long sequences. [25] used LSTMs to build a relatively simple neural translator that achieved results comparable to those of state-of-the-art SMT systems. Another type of gated units are GRUs, developed in [7] for a phrase scoring system based on Neural Networks, as a part of a Phrase-Based translator. They are simpler than regular LSTM units, they have fewer gating units, thus reducing their training time. They, however, have been shown to be as powerful as LSTM. 2.2 Neural Machine Translation 9 RESET GATE 1− xtht−1 rt ht−1 zt tanh ⊙ ht−1 xt ~ ht ht−1 ht UPDATE GATE xt ⊙ ⊙ + Figure 2.3: Illustration of the Gated Recurrent Unit (GRU). The reset gate adjusts how much of the previously stored information is retained and the update gate adjusts how much of the newly acquired (current iteration) information is kept. xtand htfollow the same notation as in Figure 2.2, while ˜ htrepresents the updated state at iteration t,rtis the output of the reset gate at iteration tand ztis the output of the update gate at iteration t. At a given time t, a GRU cell holds a hidden state ht. Given its hidden state in the last iteration (at t−1), its current updated state ˜ htand the output of the update gate zt, the current hidden state will be computed as follows: ht= (1−zt)ht−1+zt˜ ht(2.5) where stands for the element-wise multiplication. The updated state ˜ htcan be computed from the current input xt, the previous hidden state ht−1and the output of the reset gate rt: ˜ ht=tanh(Wxt+U[rtht−1]) (2.6) where Wand Uare weight matrices, parameters of the model. Bias terms have been left out for readability. Finally, the output of the reset and update gates are computed following these formulae: rt=σ(Wrxt+Urht−1)(2.7) zt=σ(Wzxt+Uzht−1)(2.8) where Wrare Urreset gate weight matrices, Wzand Uzare update gate weight matrices and σis the element-wise logistic function. 10 State of the art x1 hf 1 y1 xt hf t−1 yt ... hf t Input layer Forward layer Output layer xJ hf J−1 yJ ... ... ... hb 2 hb t hb t+1 hb J Backward layer hb J hf J hb t hf t hb 1 hf 1 Figure 2.4: Structure of a BRNN. 2.2.2. Bidirectional Recurrent Neural Network In order to improve the quality of the translations, when it comes to processing a given word in the middle of a sentence, we can choose to examine not only the words that precede but also the ones that follow it. In order to do that, we need to process the sentence both ways: forward and backwards. The current architecture of RNN that has been introduced only accounts for previous words, but we can include an additional hidden layer, independent of the previous one, that will process words from last to first. This architecture, introduced in [24] as Bidirectional Recurrent Neural Network (BRNN), will allow the network to take decisions based on the whole sentence context. In this architecture, there are two hidden states: forward (hf) and backward (hb), that are computed as follows: hf t=fh(xt,ht−1)(2.9) hb t=fh(xt,ht+1)(2.10) yt=fo(hf t,hb t)(2.11) 2.2.3. Encoder-Decoder model In the model proposed by Cho et al. (2014) and Sutskever et al. (2014), we have a system composed of two RNNs: an Encoder and a Decoder. The model used in this work is the one proposed by Bahdanau et al. (2014), which is an extension of the aforementioned two. 2.2 Neural Machine Translation 11 The Encoder, which in our case is a BRNN, encloses the input sentence in a context vector. By means of reading each element of the input sentence, the hidden state of the network changes, and after completing the reading it is a compendium of the sentence. In our case, the context vector is obtained from the concatenation of the hidden states of the forward recurrent layer and the backward recurrent layer. The Decoder in turn has the task to generate the output sentence from the aforementioned context vector. Iteratively, the Decoder will generate a word given the context vector and the previously generated words. Once the Decoder generates a special “end of line” word, the complete sentence is the system’s hypothesis for the source sentence. For each word xj(1 ≤j≤J) in the source sentence, belonging to the source vocabulary Vs, we produce a vector xj∈ {0, 1}Vs, where every entry is set to zero except the one corresponding to xj, which is set to one. This is called one-hot codification. Then, the words are projected to a fixed-size continuous vector in the following way: xj=Esxj(2.12) where xjis the embedding of word xjand Esis the source language projection matrix. The sequence of word embeddings, represented as x=x1, ...,xJ, is the input of the Encoder. After processing each word xj, the hidden state of the Encoder hjis recorded. Since we have opted for using a BRNN as the Encoder, our hidden state is actually hj= [hf| j;hb| j]|(2.13) Once the sentence has been fully processed, an iterative process begins. At each step, a non-linear function qis applied to the sequence of hidden states and to the hidden state of the Decoder network at the previous step, in order to obtain a context vector c. In this work an attention mechanism has been used, therefore the function qis a weighted sum of the hidden states. This works like an alignment model, implemented by a Multilayer Perceptron (MLP), between the source sentence and the target sentence, and thus, we have a different context vector cifor each step i: ci=q({h1,...,hJ},gi−1)(2.14) Thus, ciis a dynamic representation of the input sentence, based in the state of the Decoder. This context vector is then fed to the Decoder. The Decoder processes the context vector, and outputs Vsreal numbers between zero and one, where the i-th output represents the probability of the i-th word in the target language to be next in the translation of the source sentence. This output depends on ci,giand the word embedding representation of the last emitted word. This is implemented through a softmax layer, which ensures that all the probabilities add up to one: y0 k=yk ∑Vs l=1yl (2.15) where yirepresents the i-th input of the softmax layer, and y0 irepresents its i-th output, always between zero and one. Therefore, the probability of a word at time-step iwould be: p(yi|y1, ..., yi−1,x;θ) = y| iϕ(Vη(yi−1,gi,ci)) (2.16) 12 State of the art where ϕ(·)is a softmax function, yiis the one-hot vector representation of word yi,Vis the weight matrix and ηis the output of a RNN with GRU units and a maxout output layer[12]. 2.2.4. Training Following Equation (2.1), our network aims to approximate P(y|x)in the following way: P(y|x) = I ∏ i=1 P(yi|y1, ..., yi−1,x)(2.17) In order to maximize P(y|x)for our training set, consisting of a bilingual corpus of S sentence pairs, and according to Equation (2.17), we need to find a set of parameters for our model ˆ θsuch as: ˆ θ=argmax θ S ∏ s=1 I ∏ i=1 p(y(s) i|y(s) 1, ..., y(s) i−1,x(s);θ) =argmax θ S ∑ s=1 I ∑ i=1 log(p(y(s) i|y(s) 1, ..., y(s) i−1,x(s);θ)) (2.18) where x(s)and y(s)represent the s-th sentence of the training set in the source and target languages respectively, and Iis the length of the s-th target sentence. Since each word of the system’s hypothesis depends on previously generated words and the context vector, and the context vector depends only on the source sentence, both Encoder and Decoder can be trained as a whole to maximize the conditional probability of the target sentences given the source sentences. So far, we have introduced the knowledge field of Machine Translation. We have outlined the approaches that have been adopted in the last decades to advance the quality of translators, and we have described state-of-the-art techniques that power the top translation software in the field. Finally, we have reviewed in depth the Encoder-Decoder approach and the Neural Networks employed in it. What follows is a description of the work that was carried out in order to perform the experiments. CHAPTER 3 Online learning 3.1 Training Neural Networks Our objective in training is to maximize the log-likelihood of the data we use for training, in an attempt to produce a system that can generalize that set of translations into the overall translation task. By following Equation (2.18), we can tune the system parameters θto maximize this sum of log-probabilities. The trainable parameters of RNNs are the weight matrices. Most Neural Network training algorithms are iterative, they update the network weights according to a rule step by step, until a given condition is accomplished. Update rules, step size and stopping condition define the different learning techniques. The following is a classification by step size[19]: •Batch learning techniques are those that update the weights of the network after the whole training set has been processed. Once the system has evaluated every sample, its weights are updated to fit those, and a new training iteration begins. If the termination condition is reached, the training stops. •Mini-batch learning techniques indicate that the updates must be applied after an arbitrary number of samples have been processed. •Online learning techniques, finally, are those that update the network weights after every single sample. They are a particular interest of us, on account of them being a perfect fit for the situation described at the beginning of this work: improving a system after a new sample is obtained. With the goal of enhancing a neural translator with its use, a setup like the following can be adopted: the system generates translations for a human translator, who inputs corrected versions of those translations, which the system uses one by one to update the weights of its internal Neural Network. 3.2 Online learning algorithms In the next sections we describe the algorithms we have chosen to compare. 13 14 Online learning 3.2.1. Stochastic Gradient Descent SGD is a learning algorithm[22] that approximates Gradient Descent by updating the weights of the system using the following rule: θs=θs−1−η∇`s(θs−1)(3.1) This update is performed with the gradient for each sample of the training set, thus approximating the gradient of the whole set. In (3.1) ηis the learning rate, which sets the pace of the updates to the model. This is the only parameter we can tune in this algorithm. It is usually set to values lower than one, in order to modify the model in small steps towards minima in the error function. Every time the weights of the network are updated, the system tries to improve its performance towards the new sample, and in the process, its performance with previously seen samples may get worse. In order to try to achieve a good performance in the target data of the system as a whole, the learning rate is used. If the learning rate were too high, the system would aggressively try to fit new samples at the expense of past data, resulting likely in an overall bad performance. If it were too small, the network would conservatively learn the data, requiring a very high number of iterations, and thus a very long time, to be trained. 3.2.2. AdaGrad AdaGrad is a family of adaptive, subgradient, online learning algorithms developed in [11], based on SGD, that is expected to outperform it for high-dimensional, sparse features. The implementation used in this work is an approximation obtained from [13]. Its update rule is as follows: vs=vs−1−ηG−1/2 s∇v`(3.2) Gs=Gs−1+ (∇v`)2(3.3) where vis any given weight of θ,∇v`represents the gradient of the loss function with the previous weight set with respect to weight vbefore processing sample sand Gsrepresents the sum of squared gradients before processing sample s. At any given time, Gs=∑s i=1∇v`2 i. In this case, we also have a learning rate parameter that we can tune in order to seek the optimal performance of the algorithm. 3.2.3. Passive-Aggressive Passive-Aggressive are a family of margin-based online learning algorithms, proposed in [8]. The goal of those algorithms is to find, at each step, the model which, being as close as possible to the current one, achieves some given margin on the current sample. This is a constraint optimization problem that is solved by the Lagrange multipliers technique to find an update rule that meets the conditions. Since the margin requirement might be a hard one, the PA-II and PA-III algorithms include an “aggressiveness” hyperparameter that allows for a trade-off between the desired margin and the proximity to the current model. 3.2 Online learning algorithms 15 These algorithms have the following update rule: θs+1=θs+sign(ys−ˆys)τsxs(3.4) where τsdepends on the particular algorithm: τs=       `s ||xs||2, PA min(C,`s ||xs||2), PA-I `s ||xs||2+1 2C , PA-II (3.5) where Cis a parameter called aggressiveness in [8] and `sis the value of the loss function at time t. As we can see in Equation (3.4) and Equation (3.5), PA-I has no hyperparameters and PA-I and PA-II have one: C. In order to solve ˆ θ=argminθ1 2||θ−θs||2s.t. `(θ,xs,ys,hs)≤0 we use the Lagrange multipliers technique: `(θ,xs,ys,hs) = log pˆ θ(hs|xs)−log pˆ θ(ys|xs)(3.6) We start by obtaining Lagrange function: L(θ,λ) = 1 2||θ−θs||2+λ`(θ,xs,ys,hs)(3.7) where λis a Lagrange multiplier. Next we obtain the gradient, which would be zero at the minimum: ∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0 (3.8) θ=θs−λ∇θ`(θ,xs,ys,hs)(3.9) Afterwards, we get the pseudo-dual function: LD(θ,λ) = 1 2λ2||∇θ`(θ,xs,ys,hs)||2+λ`(θ,xs,ys,hs)(3.10) As we did before, we look for the minimum: ∂LD(θ,λ) ∂λ =λ||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0 (3.11) ˆ λ=−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2(3.12) Which is the optimal solution for the Lagrange multiplier λ. Along with Equation (3.9), we can obtain the pseudo optimal solution: θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2(3.13) PA-I requires solving ˆ θ=argminθ1 2||θ−θs||2+Cξs.t. `(θ,xs,ys,hs)≤ξ, which is done similarly to equations 3.7 to 3.13: 16 Online learning L(θ,λ1,λ2) = 1 2||θ−θs||2+Cξ+λ1(`(θ,xs,ys,hs)−ξ)−λ2ξ ∇θL(θ,λ1,λ2) = θ−θs+λ1∇θ`(θ,xs,ys,hs) = 0 θ=θs−λ1∇θ`(θ,xs,ys,hs) ∂L(θ,λ1,λ2) ∂ξ =C−λ1−λ2=0→C=λ1+λ2 LD(θ,λ1,λ2) = 1 2λ2 1||∇θ`(θ,xs,ys,hs)||2+Cξ +λ1`(θ,xs,ys,hs)−(λ1+λ2)ξ ∂LD(θ,λ1,λ2) ∂λ1=λ1||∇θ`(θ,xs,ys,hs)||2+`(θ,xs,ys,hs) = 0 ˆ λ1=min(C,−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2) θ=θs−min(C,−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2)∇θ`(θ,xs,ys,hs)(3.14) whereas PA-II requires solving ˆ θ=argmin θ 1 2||θ−θs||2+Cξ2s.t. `(θ,xs,ys,hs)≤ξ L(θ,λ) = 1 2||θ−θs||2+Cξ2+λ(`(θ,xs,ys,hs)−ξ) ∇θL(θ,λ) = θ−θs+λ∇θ`(θ,xs,ys,hs) = 0 θ=θs−λ∇θ`(θ,xs,ys,hs) ∂L(θ,λ) ∂ξ =2Cξ−λ=0→ξ=λ 2C LD(θ,λ) = 1 2λ2||∇θ`(θ,xs,ys,hs)||2+Cλ 2C2 +λ`(θ,xs,ys,hs)−λλ 2C ∂LD(θ,λ) ∂λ =λ||∇θ`(θ,xs,ys,hs)||2+λ 2C+`(θ,xs,ys,hs)−λ C=0 ˆ λ1=−`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2−1 2C θ=θs+`(θ,xs,ys,hs)∇θ`(θ,xs,ys,hs) ||∇θ`(θ,xs,ys,hs)||2−1 2C (3.15) As can be seen in Equation (3.13), θis found in both sides of the equation, thus the need for the approximation using fixed-point iterators, which can be seen in Algorithm 3.1 for the PA algorithm, while PA-I and PA-II are identical, requiring only a change in the update line for the corresponding formula, to be like Equation (3.14) and Equation (3.15). Although the implementation is further explained in Section 4.1.2, it is worth noting that in the preliminary experiments no promising results were achieved with either of the three versions of the algorithm, and therefore it was dropped from the experimentation plan towards the end of the project. 3.2 Online learning algorithms 17 Input: Parameters at the beginning of the iteration θs Source sentence xs Target sentence ys Hypothesis hs Output: θnew Initialization: θnew =θs repeat 1. θold =θnew 2. θnew =θs+`(θold,xs,ys,hs)∇θ=θold `(θ,xs,ys,hs) ||∇θ=θold `(θ,xs,ys,hs)||2 until θold == θnew Algorithm 3.1: Passive-Aggressive approximation pseudocode. 24 Experiments Iterations Learning rate 0.05 0.1 0.2 0.4 0.8 1 66.2 66.0 65.9 64.2 61.0 3 66.0 66.2 - - - 5 66.4 66.2 65.3 60.9 - 10 66.5 66.4 66.1 60.8 - 20 66.6 66.8 - - - Table 4.3: Hyperparameter search for the Xerox task, from English to Spanish, with SGD algorithm. Iterations Learning rate 0.05 0.1 0.2 0.4 0.8 1 69.9 69.9 69.8 68.7 63.7 370.1 69.8 70.6 68.4 61.3 5 70.5 70.0 69.5 69.3 60.0 Table 4.4: Hyperparameter search for the Xerox task, from Spanish to English, with SGD algorithm. SGD Table 4.3 shows general but small improvement using a learning rate lower than 0.2. It is possible that slightly better results could be achieved with even lower learning rates, but those experiments fell outside the scope of this work. The set of hyper parameters (0.05,5)was selected as a trade-off between performance and quality, since the best result (66.8) was achieved performing 20 iterations per sample, setup which would be liable of slowing down too much translation software. Table 4.4 nonetheless showed much better results than Table 4.3, arguably because of differences in the models (different number of hidden nodes in their networks, different training time...), or maybe because this task was easier than the former (is translating from English to Spanish harder than from Spanish to English?). The best result in this batch of experiments achieved an improvement of 5.3 points in the BLEU score. AdaGrad According to Table 4.5, the model in the English-Spanish task improved the most by performing 20 training iterations per sample with a learning rate of 0.0001, and just as well by performing 3 iterations per sample with a learning rate of 0.0005. Since speed Iterations Learning rate 5e-5 1e-4 5e-4 1e-3 1 67.5 67.7 69.0 67.7 367.7 68.2 69.4 68.0 5 67.8 68.8 69.3 68.3 10 68.3 69.1 68.6 67.6 20 68.9 69.4 68.9 66.6 Table 4.5: Hyperparameter search for the Xerox task, from English to Spanish, with AdaGrad algorithm. 4.3 Hyperparameter search 25 Iterations Learning rate 5e-5 1e-4 5e-4 1e-3 5e-3 1 69.9 69.8 71.0 70.8 59.8 3 69.8 70.2 71.1 - - 569.8 70.1 71.7 70.5 57.6 Table 4.6: Hyperparameter search for the Xerox task, from Spanish to English, with AdaGrad algorithm. Iterations Learning rate 0.05 0.1 0.2 0.4 0.8 1 35.5 35.3 35.4 34.8 33.3 335.3 35.5 36.1 35.6 35.4 5 35.4 35.6 35.8 36.1 - Table 4.7: Hyperparameter search for the EU task, from English to Spanish, with SGD algorithm. is important in the context of this work, the latter has been chosen as the optimal set of hyperparameters for AdaGrad in this task. It is worth noting that none of the results of this table result in a decline in the performance. Moreover, the average improvement was of 2.3 percentage points, which shows that AdaGrad achieved a much better overall improvement in this set than SGD. Nevertheless, this task is being used for hyperparameter search, so conclusions must not be drawn from this comparison, given that we purposely select the best result of each of them. In Table 4.6 we see improvements of the same order of Table 4.4, which leads to think that this task is quite adept at being retrained. 4.3.2. EU task The EU corpus[16] was obtained from the Bulletin of the European Union, which is publicly available in all the official languages of the European Union. As in the Xerox tasks, only Spanish to English and English to Spanish were used. The corpus was tokenized as well, but not transformed to lower case as we did in Section 4.3.1. SGD Table 4.7 shows a case not encountered so far: none of the results show an improvement in the quality of translations. There are several factors that can contribute to this phenomenon. First of all, this set is much smaller than that of the previously shown tasks: 400 sentence pairs versus 1012 in Xerox. Since the BLEU score is a measure that attempts to match human judgement when averaged over a corpus[21], its value when the set is small can be expected to be less reliable than the case where it is applied to a big corpus. Furthermore, the domain of the EU corpus is in all likelihood more complex than the domain of Xerox. Xerox corpus is full of short sentences, with very repetitive words, like printer features and options. Numbers in Xerox are most of the time model identifiers, which do not change. EU bulletin is very diverse, has long sentences, a high quantity of numbers of records, dates, percentages, file sizes, and so on. Also, the initial BLEU in both could not be any more different, and the vocabulary size in both languages in Xerox is less than half than in the EU models. 26 Experiments Iterations Learning rate 0.05 0.1 0.2 0.4 0.8 1 35.1 35.1 35.1 35.3 34.4 335.0 34.8 35.2 36.0 35.5 5 35.1 35.7 35.8 35.8 34.8 Table 4.8: Hyperparameter search for the EU task, from Spanish to English, with SGD algorithm. Iterations Learning rate 5e-5 1e-4 5e-4 1e-3 1 35.9 35.4 35.7 35.0 3 36.1 35.7 36.1 35.5 535.8 36.2 35.7 33.7 Table 4.9: Hyperparameter search for the EU task, from English to Spanish, with AdaGrad algorithm. We can say without a shade of doubt that the Xerox translation models in this work are more effective than their EU counterparts. It remains a question whether this makes the task of retraining easier or harder, or whether it is inmaterial to it. Further research would be necessary to answer it. Finally, given the assumptions of correlation between model and algorithm performances in different tasks that were issued in the introduction of this work, we must assume too that the experiment with the test set that involves this particular configuration will deteriorate the model performance, but not as much as with the other parameters that were tried. A small improvement can be observed in some of the entries of Table 4.8. Even those setups that resulted in a decline in quality of translations did so only by a very small amount. This could be attributed to the small size of the development set, in comparison to the other corpus. AdaGrad The results in Table 4.9 appear to be very similar to those in Table 4.7: no improvement in any case. This can likely be attributed to the same hypothetical reasons that were given for the results in Table 4.7. The best result of the table was chosen for furhter experimentation. Table 4.10 shows slightly more promising results than Table 4.8, even though slightly fewer cases were attempted. Iterations Learning rate 5e-5 1e-4 5e-4 1e-3 1 35.3 35.7 35.8 35.0 3 35.5 36.1 35.4 35.5 535.7 35.9 36.3 35.8 Table 4.10: Hyperparameter search for the EU task, from Spanish to English, with AdaGrad algorithm. 4.4 Test set experiments 27 Task BLEU Initial SGD AdaGrad Xerox En-Es 55.2 55.9 57.1 Xerox Es-En 46.1 51.7 50.4 EU En-Es 36.8 36.4 36.4 EU Es-En 35.7 34.8 36.0 Table 4.11: BLEU score obtained in the experiments on the test set of each task, both without retraining and by retraining with each algorithm using the hyperparameters selected in Section 4.3. Xerox En-Es Xerox Es-En EU En-Es EU Es-En −1 0 1 2 3 4 5 6 0.7 5.6 −0.4 −0.9 1.9 4.3 −0.4 0.3 ∆BLEU SGD AdaGrad Figure 4.1: Differences in BLEU obtained in the test set of each task by retraining with each algorithm, using the hyperparameters selected in Section 4.3. The values used can be found in Table 4.11. 4.4 Test set experiments From the data in Table 4.11 we see that AdaGrad outperforms SGD in two of our four cases, and matches it in another one. The data, however, is insufficient to draw conclusions about AdaGrad being in general better for this scenario. SGD outperformed AdaGrad in Xerox Spanish-English by a relatively large margin, in comparison with the other results. The improvement achieved by each algorithm on each model is plotted in Figure 4.1. Both algorithms showed considerably better results in the Xerox tasks than in the EU ones, which could be attributed to, among other things, the task complexity and/or the already-existing model performance. The models that obtained a high BLEU initially improved their results, they probably took profit of the data and enhanced their parameters as to better fit the translation task. The models that obtained a relatively low BLEU in the initial measurement may have not been able to assimilate the new data and may have 28 Experiments Xerox En-Es Xerox Es-En EU En-Es EU Es-En 0 10 20 30 40 50 60 Execution time / sentence pairs (s) SGD AdaGrad Figure 4.2: Execution time of experimens in the test sets. been negatively affected by the parameter updates (as they may have in the experiments with the development set), but even then, the quality of the hypotheses was not much worse off, and AdaGrad algorithm could achieve an improvement in the EU Es-En task The decline in performance observed by retraining in both EU tasks in previous experiments is consistent with these results. The Xerox English-Spanish translation task has produced much different results than prior experiments, obtaining much lower improvements, while Xerox Spanish-English has maintained a high degree of improvement with both algorithms. In Figure 4.2 we can observe the different execution times of each algorithm, averaged over the number of sentence pairs of the relevant set. We can observe that the execution time for EU tasks was much longer than for Xerox tasks, likely due to their longer sentences and higher vocabulary size (which makes their models even larger). CHAPTER 5 Conclusions and future work We have developed a full-fledged experimentation environment for online adaptation of neural translators, able to perform automated batches of experiments with proper handling and storing of the results. We have selected and processed data for experiments, and performed several of them with different sets of hyperparameters, and have found the ones among them that work best for each combination of task and algorithm. With a different set but for the same task, we have performed experiments with both algorithms configured to their empirically-found best capacities and have obtained a comparison of how the algorithms can improve or worsen the quality of neural translators. We have obtained an iterative approximation of the Passive-Aggressive family of online learning algorithms, produced an implementation for our experimentation framework and observed the lack of promising results. Due to this, no further experimentation was carried out with those algorithms. In three out of four translation tasks, we observed a slight correlation between the results in the development set and the results in the test set. AdaGrad algorithm resulted more promising than SGD, but the latter outperformed the former in one task. All things considered, we cannot state that AdaGrad would be the best choice for every task, but a priori it is a better candidate. Several questions remain unanswered, such as to why is there such disparity in the results between development set and test set in the Xerox English-Spanish task, or why the EU tasks put up such a challenge against both algorithms. Further research is required to answer those questions. More experiments, involving more datasets and more language pairs are required, since neural translators are usually trained and used for single language pairs, and the results observed using one of them may not correspond at all with results obtained from a different one. In this work we have used two corpora and one language-pair (in both directions), and even then we have found significant differences between tasks (especially between both directions of the Xerox English to/from Spanish tasks). The size of the EU development set, especially when compared to that of the Xerox corpus, raises the question of whether the hyperparameter search for the EU tasks may have been compromised, or at the least whether the size of this set has handicapped the hyperparameter search for those models. More similar pairs of sets could be used to help dispel this doubt. Finally, the approximation of Passive-Aggressive algorithms must be revised in search of alternative methods to fixed point iterators, to eventually test them against the challenges presented in this work, and compare them to the other two algorithms we have used. 29 Acronyms A. Acronyms BLEU . . . . . . . . . . . BiLingual Evaluation Understudy BRNN . . . . . . . . . . . Bidirectional Recurrent Neural Network CBS . . . . . . . . . . . . Corpus-Based System GRU . . . . . . . . . . . . Gated Recurrent Unit LSTM . . . . . . . . . . . Long Short Term Memory MLP . . . . . . . . . . . . Multilayer Perceptron MT . . . . . . . . . . . . Machine Translation NLP . . . . . . . . . . . . Natural Language Processing NMT . . . . . . . . . . . Neural Machine Translation RBS . . . . . . . . . . . . Rule-Based System RNN . . . . . . . . . . . Recurrent Neural Network SGD . . . . . . . . . . . . Stochastic Gradient Descent SMT . . . . . . . . . . . . Statistical Machine Translation 31 Bibliography [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. [2] Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, and Yoshua Bengio. Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590, 2012. [3] Jerome R Bellegarda. A latent semantic analysis framework for large-span language modeling. In EUROSPEECH, 1997. [4] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155, March 2003. [5] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157– 166, 1994. [6] M Asunción Castano, Francisco Casacuberta, and Enrique Vidal. Machine translation using neural networks and finite-state models. Theoretical and Methodological Issues in Machine Translation (TMI), pages 160–167, 1997. [7] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. [8] Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online passive-aggressive algorithms. J. Mach. Learn. Res., 7:551–585, December 2006. [9] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In ACL (1), pages 1370–1380. Citeseer, 2014. [10] Bonnie J Dorr, Pamela W Jordan, and John W Benoit. A survey of current paradigms in machine translation. Advances in computers, 49:1–68, 1999. [11] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. [12] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv:1302.4389, 2013. 33