A deep learning approach for automatically generating descriptions of images containing people

A deep learning approach for automatically generating descriptions of images containing people Trabajo de Fin de Grado Curso 2017–2018 Autor Marta Aracil Mu˜noz Directores Gonzalo M´endez Pozo Raquel Herv´as Ballesteros Grado en Ingenier´ıa Inform´atica Facultad de Inform´atica Universidad Complutense de Madrid A deep learning approach for automatically generating descriptions of images containing people Trabajo de Fin de Grado en Ingenier´ıa Inform´atica Departamento de Ingenier´ıa del Software e Inteligencia Artificial Autor Marta Aracil Mu˜noz Directores Gonzalo M´endez Pozo Raquel Herv´as Ballesteros Convocatoria: Septiembre 2018 Grado en Ingenier´ıa Inform´atica Facultad de Inform´atica Universidad Complutense de Madrid Abstract Generating image descriptions is a challenging Artificial Intelligence problem with many interesting applications such as robots’ communication or helping visually impaired people. However, it is a complex task for computers: it requires Computer Vision algorithms, to understand what the image depicts, and Natural Language Processing algorithms, to generate a well-formed sentence. Nowadays, deep neural networks are the state-of-the-art in these two Artificial Intelligence fields. Furthermore, we believe that images that contain people are described in a slightly different manner and that restricting an image description generator model to these images may produce better descriptions. Therefore, the main objective of this project is to develop a Deep Learning model that automatically produces descriptions of images containing people and to conclude if it is a good practice the restriction to this kind of images. For this purpose, we have reviewed and studied the literature in the field and we have built, trained and compared four different models using Deep Learning techniques and a GPU to speed-up the computation, as well as a big and complete dataset. Keywords Deep Learning, Computer Vision, Natural Language Processing, image description generation, Keras, GPU, dataset. v Resumen Generar descripciones de im´agenes es un problema de Inteligencia Artificial con muchas aplicaciones interesantes como la comunicaci´on de robots o ayudar a personas con discapacidad visual. Sin embargo, es una tarea compleja para un ordenador: requiere algoritmos de visi´on por computador para entender lo que la imagen representa y algoritmos de procesamiento de lenguaje natural para generar una frase bien formada. Hoy en d´ıa, las redes neuronales profundas son el estado del arte en estos dos campos de la Inteligencia Artificial. Por otra parte, creemos que las im´agenes que contienen personas se describen de manera ligeramente diferente y que restringir un modelo de generaci´on de descripciones de im´agenes a im´agenes de este tipo puede producir mejores descripciones. Por lo tanto, el principal objetivo de este proyecto es desarrollar un modelo de aprendizaje profundo que produce autom´aticamente descripciones de im´agenes que contienen personas y concluir si es una buena pr´actica la restricci´on a esta clase de im´agenes. Para ello, hemos revisado y estudiado la literatura y hemos construido, entrenado y comparado cuatro modelos diferentes usando t´ecnicas de aprendizaje profundo y una GPU para acelerar los c´alculos, as´ı como un dataset grande y completo. Palabras clave Aprendizaje profundo, visi´on por computador, procesamiento de lenguaje natural, generaci´on de descripciones de im´agenes, Keras, GPU, dataset. vii Contents 1 Introduction and objectives 1 1.1 Objectives..................................... 2 1.2 Documentstructure ............................... 3 2 Introducci´on y objetivos 5 2.1 Objetivos ..................................... 6 2.2 Estructura del documento . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 State of the art 9 3.1 Traditional algorithms for image analysis . . . . . . . . . . . . . . . . . . . 10 3.2 Traditional approach to NLG . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3 MachineLearning................................. 11 3.4 DeepLearning .................................. 13 3.4.1 Feedforward Neural Networks . . . . . . . . . . . . . . . . . . . . . . 14 3.4.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . 15 3.4.3 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 17 3.5 Image description generators . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4 Framework 21 4.1 GPU........................................ 21 4.2 Deep Learning framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.3 Visualization: TensorBoard . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 Imagedatasets .................................. 25 4.4.1 Datasetelection ............................. 28 ix 4CHAPTER 1. INTRODUCTION AND OBJECTIVES Chapter 2 Introducci´on y objetivos En nuestra condici´on de humanos, constante e inconscientemente hacemos descripciones del mundo que nos rodea para comunicarnos con la gente. Describimos lo que vemos en muchas situaciones diferentes, como por ejemplo cuando damos indicaciones a alguien sobre c´omo ir a alg´un sitio (sigue recto hasta que veas el edificio rojo y despu´es gira a la derecha a la calle que tiene los ´arboles grandes y una fuente), cuando le contamos una historia a un amigo (estaba de compras cuando me encontr´e a Sara de la mano de un se˜nor mayor alto) o cuando subimos una foto a Internet (¡Pas´andomelo genial en la playa con mi vestido azul nuevo!). Mientras que estas descripciones las hacemos de manera sencilla y natural, sin tener que pensarlo demasiado, generar autom´aticamente descripciones de lo que hay en una imagen es una tarea muy compleja y exigente para un ordenador. Generar una buena descripci´on de una imagen requiere primero analizar y entender lo que aparece en ella y despu´es generar una frase en lenguaje natural que explique esos elementos, est´e bien formada y sea concisa. Por lo tanto, generar descripciones de im´agenes es m´as dif´ıcil que las t´ıpicas tareas del campo de visi´on por computador, como son la clasificaci´on de im´agenes o la detecci´on de objetos, ya que conlleva no solo t´ecnicas de Visi´on por Computador (CV), sino tambi´en Procesamiento del Lenguaje Natural (NLP); dos campos de la Inteligencia Artificial que por lo general han llevado caminos separados y no se suelen estudiar ni aplicar en conjunto. La generaci´on de descripciones de im´agenes se puede considerar como un caso particular de la traducci´on autom´atica donde, en lugar de traducir una frase dada a otro idioma, hay que traducir una imagen de entrada a su descripci´on. La imagen de entrada debe ser codificada a un vector de representaci´on y, despu´es, esta representaci´on debe ser decodificada a una frase en lenguaje natural. Para lograr esto, podemos combinar resultados del estado del arte de ambos campos: los resultados de Visi´on por Computador para obtener el vector de representaci´on y los de NLP (en particular, el estado del arte de la Generaci´on de Lenguaje Natural (NLG)) para construir la descripci´on. Este problema tan interesante de Inteligencia Artificial ha despertado el inter´es de grandes empresas como Facebook o Google. El problema de describir im´agenes au5 6CHAPTER 2. INTRODUCCI ´ ON Y OBJETIVOS tom´aticamente no solo es atractivo por su inter´es acad´emico y cient´ıfico, sino tambi´en por sus muchas aplicaciones. La principal y m´as importante aplicaci´on es generar descripciones para las personas con alguna discapacidad visual, bien sea para ayudarles en su d´ıa a d´ıa (por ejemplo, para moverse por la ciudad o comprar en un supermercado), para acercarles a la cultura generando descripciones de los cuadros de un museo o, como vivimos en un mundo claramente dominado por fotos, para hacer m´as accesibles las redes sociales, y las p´aginas de Internet en general. Pero tambi´en tiene otras aplicaciones como por ejemplo en rob´otica para la comunicaci´on de los robots (transformando en frases lo que ven e identificando con qui´en est´an hablando) o en medicina para describir im´agenes m´edicas. Por otro lado, en los ´ultimos a˜nos hemos podido presenciar un incre´ıble aumento del inter´es en obtener informaci´on de los datos y estamos actualmente en lo que llaman la era data-driven (dirigida por los datos). Cada d´ıa, se crean millones de nuevos datos, muchas empresas tienen recogida una gran cantidad de datos y quieren sacarles provecho, descubriendo patrones ocultos y perspectivas para obtener buenas decisiones de negocio. Debido a esta gran cantidad de datos disponibles y al crecimiento de la capacidad computacional para realizar c´alculos complejos, las t´ecnicas de aprendizaje autom´atico y aprendizaje profundo se han podido desarrollar y estudiar convirti´endose en el estado del arte en muchos campos, lo que ha hecho que est´en en boca de todos. Por lo tanto, este trabajo busca aunar estos dos conceptos tan populares y atractivos, desarrollando un generador de descripciones de im´agenes utilizando t´ecnicas de aprendizaje profundo. 2.1 Objetivos Respecto a las descripciones de im´agenes, Bernardi et al. [Bernardi et al., 2016] distinguen dos enfoques principales para su generaci´on: modelos que generan descripciones nuevas para una imagen de entrada y modelos que construyen la descripci´on rellenando plantillas o usando descripciones asociadas a im´agenes similares; nosotros centraremos nuestro trabajo en los primeros. Vamos a limitar el alcance de este trabajo a generar descripciones de im´agenes de personas, ya que consideramos que es un enfoque interesante y podremos centrar todos los esfuerzos en ello. El trabajo relacionado en el campo de la generaci´on autom´atica de descripciones de im´agenes no se suele centrar en un tema en concreto, tratando de abarcar todas las posibles descripciones e im´agenes y no le han dado demasiada atenci´on a las im´agenes de personas. Consideramos que no describimos de la misma manera a una persona que a otros elementos de una imagen, y por tanto, al restringirnos a im´agenes de personas podemos obtener mejores descripciones y resultados. Con todo esto, los objetivos principales de este trabajo son: •Revisar y analizar la bibliograf´ıa y el estado del arte de las t´ecnicas de generaci´on 2.2. ESTRUCTURA DEL DOCUMENTO 7 de descripciones de im´agenes. •Estudiar y entender los modelos de aprendizaje profundo y sus aplicaciones. •Estudiar los diferentes entornos de trabajo y herramientas enfocadas al aprendizaje profundo. •Desarrollar diferentes modelos que generen autom´aticamente descripciones de im´agenes de personas y tratar de reproducir los resultados del estado del arte. •Evaluar y comparar los distintos modelos creados y determinar cu´al es el mejor y si es satisfactorio centrar los esfuerzos y limitar nuestros modelos a caracter´ısticas de personas. 2.2 Estructura del documento La estructura de este documento ser´a la siguiente: el Cap´ıtulo 3 da una explicaci´on muy detallada del estado del arte tanto en el campo de la Generaci´on de Lenguaje Natural como en el de Visi´on por Computador, en este cap´ıtulo tambi´en se detallan algunos generadores de descripciones de im´agenes. En el Cap´ıtulo 4 introducimos el entorno de trabajo en el que desarrollaremos este trabajo; en particular, explicamos las GPUs, el entorno cloud computing, qu´e librer´ıas para aprendizaje profundo y visualizaci´on usaremos y la elecci´on del dataset. El Cap´ıtulo 5 es el principal de este trabajo; va paso a paso por el proceso de definici´on y entrenamiento de los modelos. Explicamos en este cap´ıtulo todas las caracter´ısticas y componentes de los modelos y todas las decisiones tomadas al definirlos y entrenarlos. El Cap´ıtulo 6 est´a dedicado a mostrar los resultados obtenidos, el proceso de testing y las m´etricas m´as conocidas. Por ´ultimo, en el Cap´ıtulo 8 presentamos las conclusiones del trabajo y damos posibles l´ıneas de trabajo futuro. 8CHAPTER 2. INTRODUCCI ´ ON Y OBJETIVOS Chapter 3 State of the art Image analysis has been a challenging problem in researchers’ minds for a long time now. At first, this problem was tackled by a traditional computer vision approach, with explicit processing algorithms. Then, Machine Learning models appeared and they were the stateof-the-art in this field until 2010, when training complex deep neural networks started to become a possibility thanks to the use of GPUs computational power and the availability of more data. As of today, these neural networks, in particular Convolutional Neural Networks, are the state-of-the-art in all image analysis challenges. Similarly, the task of Natural Language Generation (NLG), has been faced by classical step-by-step algorithms, as well as Machine Learning and Deep Learning models. In particular, Recurrent Neural Networks are, as of today, the state-of-the-art in generating natural language sentences and texts. Image description combines both computer vision and natural language processing fields, and, nowadays, the state-of-the-art results are a combination of Convolutional Neural Networks, that focus on extracting image features, and Recurrent Neural Networks, to generate the sentence. In this chapter we will present previous work in the field of image analysis and Natural Language Generation, and how these fields have been addressed across time, from the point of view of traditional and explicit algorithms to the one of Deep Learning models. We start with a brief introduction to traditional algorithms for computer vision and language generation, we then explain the main characteristics of Machine Learning and we continue with Deep Learning. In the last section of this chapter, we discuss some well-known image caption generators. 9 10 CHAPTER 3. STATE OF THE ART 3.1 Traditional algorithms for image analysis Image analysis consists of the extraction of important characteristics and features of images by digital image processing techniques, and it has many different fields of application, such as robotics, security, medicine or biology. Traditional computer vision algorithms, in general, work by extracting feature vectors from images and using them to classify images. There are algorithms responsible for a particular task (noise reduction, image segmentation, corner detection, edge detection...), that then work together with other specific algorithms to carry out the whole image processing task. The use of these techniques to extract useful structural information from images, such as edges, corners or colors, reduces significantly the amount of data to be processed, as it filters out non-relevant data to focus only in the useful information extracted. The main feature detection algorithms in computer vision are: •Canny edge detector [Canny, 1986]. An edge is a sudden change in image brightness, that is, a point where the image brightness has discontinuities. These discontinuities usually correspond to changes in surface orientation, depth, material properties or scene illumination. Therefore, an edge detector may help to identify the boundaries of objects and surface markings, as well as variations in surface orientation. The Canny edge detector is one of the best edge detector algorithms, and can detect a wide variety of edges in an image. It first smooths the image by applying a Gaussian filter, then finds intensity gradients and selects the potential edges. •Harris corner detector [Harris and Stephens, 1988]. A corner is interpreted as the intersection of two edges. It is a very important image feature as it is a point invariant to translation, rotation and illumination. The Harris corner detector algorithm focuses on the detection of corners in an image, reducing the dimensionality of data to be processed and it is used in many computer vision applications such as motion tracking or stereo vision (extracting 3D characteristics from images). It has been proved to be one of the most accurate algorithms in distinguishing between edges and corners. •SIFT (Scale-Invariant Feature Transform) [Lowe, 1999]. It is a feature description algorithm used to detect and describe features in images. SIFT features are invariant to image location, scale and rotation. A descriptor vector is computed for each one of the points of interest in the image, so that they are also robust to changes in noise and illumination. This algorithm is useful for many computer vision applications, such as object recognition, motion tracking, navigation or 3D modeling. •SURF (Speeded-Up Robust Features) [Bay et al., 2006]. It is also a feature extractor and descriptor. This algorithm is a speeded-up version of SIFT. These object detection algorithms usually come after a process of image segmentation, 3.2. TRADITIONAL APPROACH TO NLG 11 that is, dividing the image into its constituents parts, and are then combined with traditional Machine Learning algorithms (SVM or K-Nearest neighbor) for image classification. 3.2 Traditional approach to NLG Natural Language Generation (NLG), involves producing understandable natural language texts from some input data. It has many different interesting applications, besides generating image descriptions, such as machine translation, robotics or textual summaries of advanced databases (e.g., financial or weather forecasts datasets) [Goldberg et al., 1994, Iordanskaja et al., 1992, Wu et al., 2016]. Traditional NLG algorithms are based on the field of formal language theory and logic rules on which we can build the language. As Reiter and Dale [Reiter and Dale, 1997] state, traditional algorithms in NLG usually follow these steps: 1. Content determination: deciding which information should be included in the output. This process creates a set of messages from the input data, and it is typically done by filtering and summarizing the data and defining the messages in some formal language, usually application-dependent. 2. Document structuring: organizing and structuring the set of messages so that the final text makes sense. The output of this step is usually represented as a tree structure. 3. Aggregation: merging related messages into sentences to compact the information. This step is not mandatory but enables fluency and readability. 4. Referring Expressing Generation: producing expressions identifying objects that the text refers to, as well as deciding about pronouns. It is related to lexicalization. 5. Lexicalization: selecting the specific words and phrases to be used to represent concepts and relations that appear in the messages. 6. Realization: applying rules of syntax, morphology and spelling to produce the final output, such as adding prepositions, making plurals or adding punctuation marks. 3.3 Machine Learning Machine Learning is the sub-field of Artificial Intelligence that studies computer algorithms that improve automatically through experience. In contrast to traditional algorithms, that have specific rules for doing the tasks they are required for, Machine Learning algorithms learn concepts without being explicitly programmed for doing so, identifying patterns from given examples to perform accurately on new, unseen data by inferring these uncovered 12 CHAPTER 3. STATE OF THE ART patterns. These algorithms can learn from and make predictions on large volumes of data, not exclusively images. The field started to be studied in the mid 1980s and early 1990s, and became more popular around 2010 as more and more data were available for learning and training the Machine Learning models. This discipline has many applications in a wide variety of fields where designing and programming explicit algorithms with good performance is difficult or infeasible. These applications include voice recognition, natural language processing, translation, search engines, computer vision, robotics, medical diagnosis, financial market analysis, advertising or recommender systems [Pang et al., 2002, Wernick et al., 2010, Bridge et al., 2014, Sarikaya et al., 2014, Baik and Bala, 2004]. We can classify Machine Learning techniques into two categories: supervised learning and unsupervised learning. In supervised learning, the Machine Learning algorithm is given a set of sample inputs together with their desired outputs, so that it can train with those examples to infer a generalized function that maps inputs into outputs and predicts well when given a new example. Therefore, already labeled-data are required in this approach. There are also some special cases of supervised learning such as semi-supervised learning, that consist of a set of examples where not all of them go with their corresponding desired output, that is, there are labeled and unlabeled data on the set; active learning, where the algorithm has only a small initial set of labeled data and it is able to interactively make queries to the user to find the desired outputs for some unlabeled data; and reinforcement learning, where there is some sort of feedback obtained from the outside in a dynamic environment as an answer for its actions. Some of the main supervised learning algorithms and approaches are the following: •Decision trees and random forests: they are used as a predictive model. In a decision tree, the branches represent the observations about an item, its features and the leaves represent class labels. Random forests [Ho, 1995] construct many decision trees to make more accurate predictions by outputting a combination of the outputs of all those decision trees, typically the mode or mean. Random forests are very popular as it is easy and clear to see and understand the decisions taken by them, while other Machine Learning algorithms are more obscure. •Logistic regression: it is a statistical method to model a binary dependent variable (classes 0 and 1) in terms of one or more explanatory variables, using the logistic function σ(z) = 1 1+e−z. This function returns the probability of belonging to class 1. The algorithm’s output is then selected by comparing the probability to a given threshold (usually 0.5). •Support Vector Machines (SVM) [Cortes and Vapnik, 1995]: it is a binary classification algorithm that given a set of points of two different classes in a n-dimensional 3.4. DEEP LEARNING 13 space, separates them with the hyperplane that is situated the furthest from all the points. This algorithm is only useful if the points are linearly separable, if not, the algorithm was improved by including kernels, that project the set of points in a bigger dimension space, to obtain a hypersurface that separates the points in the initial space. •Neural networks: it is a learning algorithm where computations are structured in connected neurons that transmit signals from one to another. Therefore, each neuron’s input is the output of another neuron (or the input of the algorithm for the first layer of neurons), and each neuron’s output is computed as a non-linear function of the input. •Nearest neighbor algorithm [Altman, 1992]: it is used both for classification (there are a discrete number of classes) and regression (the target variable is continuous), and it is based on the information of the k closest points. In classification, the output is the most common class among its k neighbors, and in regression the output is the mean of the values of these neighbors. Unsupervised learning algorithms learn from a set of unlabeled data, input examples without their desired output, inferring a function that can describe the structure in the data. Therefore, the system must be capable of finding patterns in the data in order to be able to label the new inputs; but, as the data used to train the algorithm is unlabeled, there is no way of evaluating the accuracy of the patterns found in it. The following are the most common unsupervised learning algorithms: •Clustering: these methods divide the data set into different subsets, or clusters, so that the points in the same cluster are more similar between them than between those of other groups. The goal is to maximize the similarity between points in the same subset and to maximize the difference between points of different subsets. •Principal Component Analysis (PCA) [Hotelling, 1933]: this algorithm is used to reduce the dimensionality of the data set, without losing any important information and filtering redundancy, by making linear combinations of the original data variables and selecting the ones that have the largest possible variance and are uncorrelated. 3.4 Deep Learning Deep learning is a Machine Learning technique, based on neural networks with many layers between the input and the output ones [LeCun et al., 2015]. Although the theory behind Deep Learning has been developed for many years now, deep neural networks need much computer capacity and labeled data to perform rapidly and correctly the complex tasks it is useful for. Therefore, Deep Learning has only recently 20 CHAPTER 3. STATE OF THE ART Figure 3.5: Captionbot This device is even able to learn new faces and objects if the user shows them to Horus in different angles and speaking out loud the name of the object or the person. As we have seen, more and more sophisticated image caption generators, for different purposes and contexts, are being developed, showing the interest and impact of this brandnew field and resulting in a rich literature. Chapter 4 Framework Training and developing Deep Learning models requires great computational capacity and a good election of the dataset. In this chapter, we explain what are GPUs and why are they useful for developing Deep Learning projects, which Deep Learning framework and visualization tools we have selected and which dataset we use. We developed the project using Python as programming language. 4.1 GPU GPUs (Graphics Processing Unit) were traditionally created for computer graphics and image processing to accelerate the complex calculations of real-time 2D and 3D graphics that resulted in long processing time in CPUs. GPUs are build to optimized tasks based on the SIMD parallel concept (Single Instruction Multiple Data), that is, the same instruction applied repeatedly over a bunch of data. Those repeated applications of the same instruction are all independent and can, therefore, be run simultaneously. Linear algebra operations are inherently parallel and are the basis of polygon transformations for scene rendering in computer graphics. There are other power and time consuming applications of parallel nature that could be optimized by using a GPU, but they involve reformulating the problem into graphics primitives to be able to use them, which is a very complex task. For that reason, NVIDIA developed CUDA (Compute Unified Device Architecture), a parallel computing toolkit similar to C programming language, to exploit GPUs capabilities for general purposes, which is commonly called GPGPU (General Purpose Graphics Processing Unit). Linear algebra is the basis of Deep Learning algorithms, and also many Machine Learning algorithms. Neurons within a layer apply all the same function to different input data, so they could be also parallelized and hence GPUs are always used for training these models. 21 22 CHAPTER 4. FRAMEWORK Figure 4.1: Deep Learning frameworks popularity Acquiring a GPU is not cheap, there exists cloud computing platforms, such as Amazon Web Services1, Google Cloud Platform2or Microsoft Azure3that contain different servers with several GPUs to allow users to run their models more economically. GAIA research group has built a server with an NVIDIA Titan X GPU in order to maintain Deep Learning projects for students and professors and gave us access to it, so we will use it for training this project. 4.2 Deep Learning framework CUDA has a very low level API and makes implementing Deep Learning computational graphs considerably tedious. For this reason, many higher level frameworks were created for easily describing and developing the complex tasks that Deep Learning involves. There is a wide number of frameworks available for Deep Learning training, the majority of them open-source and developed by big companies or university teams such as Microsoft, Google, Intel or Berkeley University. The most well-known and frequently used frameworks are Torch [Collobert et al., 2011], Tensorflow [Abadi et al., 2016], Caffe [Jia et al., 2014] and Theano [Theano Development Team, 2016]. Among all these frameworks, Tensorflow is probably the most popular one (as Figure 4.1, extracted from Google images, shows), used in distinguished companies such as Airbnb, Uber, Dropbox or Twitter. It is an open-source framework for advanced numerical computation, developed by the Google Brain team in 2015, replacing their propietary 1https://aws.amazon.com 2https://cloud.google.com/ 3https://azure.microsoft.com 4.2. DEEP LEARNING FRAMEWORK 23 from keras.models import Sequential from keras.layers import Dense # Building the model and learning configuration model = Sequential() model.add(Dense(units=64, activation=’relu’, input_dim=100)) model.add(Dense(units=10, activation=’softmax’)) model.compile(loss=’categorical_crossentropy’, optimizer=’sgd’, metrics=[’accuracy’]) # Training, evaluation and prediction # x_train, y_train, x_test, y_test and new_data are numpy arrays model.fit(x_train, y_train, epochs=5, batch_size=32) loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128) prediction = model.predict(new_data, batch_size=128) Figure 4.2: A simple Keras example code Machine Learning system DistBelief [Dean et al., 2012], created in 2011. Tensorflow is widely used for Deep Learning and Machine Learning and it is optimize to run models both on CPUs or GPUs. It provides an API for Python as well as for other programming languages (C++, Haskell, Java...), but the most developed one is the Python library. Tensorflow works well for modeling Convolutional Neural Networks and Recurrent Neural Networks. The basic object in Tensorflow library is the tensor, an abstraction of multidimensional arrays, being a 1-dimensional tensor a vector and a 2-dimensional tensor a matrix. Deep Learning computational graphs are build by describing tensors’ operations and gradients are calculated automatically using a specific function. However, programming in Tensorflow can be difficult and there exists a higher level API that allows flexible Tensorflow implementations in a more user-friendly manner. This API is Keras4, a high-level open-source Python library for developing Deep Learning projects more easily and readable, created in 2015 by a Google engineer and released under the MIT license. Keras works with Tensorflow, as well as Theano or CNTK [Seide and Agarwal, 2016], as backend. In 2017 Tensorflow included it as part of its library and selected it as the preferred high-level API. The basic structure in Keras is the model, a set of connected layers. Computational graphs are built by stacking layers with the add() function, where the user only needs to specify the layers and the inputs’ dimensions. Keras library provides the well-known layers but allows users to also create new ones. Configuring the training process is as simple as applying to the model the compile() function with the specific conditions (loss function, optimizer, metrics...). Training, evaluating the model and making predictions on unseen examples is done with just other high level functions: fit(),evaluate() and predict() 4https://keras.io/ 24 CHAPTER 4. FRAMEWORK respectively. Figure 4.2 shows a simple example, extracted from Keras documentation, of how to use these functions to build a neural network with two fully-connected layers. Therefore, as our needs are covered in Keras’ library, it is widely recommended for Deep Learning starters and it is more user-friendly, we decided to develop our model in Keras on top of Tensorflow. Keras and Tensorflow installation is done easily in Linux via the pip install command, creating previously a Python virtual environment. 4.3 Visualization: TensorBoard Deep Learning models are composed of millions of complex calculations that often complicate, or even make it impossible, to debug and understand the parameters and metrics, what happens during training and what is the neural network learning. For this purpose, Tensorflow developed Tensorboard5, a suite of visualization tools inside Tensorflow library that allows users to visualize computation graphs, metrics results and other useful charts. TensorBoard works by saving log files, containing execution information about the training process and the computation graph, into a specific log directory, and opening TensorBoard web server to access the graphic interface. The desired data is obtained with summary operations, tensor’s operations that produce serialized information about a model that needs to be read with TensorBoard. These summary operations receive the tensor we want to visualize and a meaningful name to distinguish all of them. Then, all the summaries created are combined using tf.summary.merge all into a single operation that generates all of them. Finally, the serialized summary is written to the specified directory in disk with tf.summary.FileWriter. To access the graphic interface of TensorBoard to visualize the summaries, we just need to go to http://localhost:6006 after typing the following command in the command line: $ tensorboard --logdir path/to/log-directory It is also possible to compare visualizations of different executions by specifying the paths to the log directories in the previous command. The appearance of TensorBoard graphic interface is shown in Figure 4.3 TensorBoard contains a wide variety of helpful visualizations: tf.summary.scalar, tf.summary.histogram,tf.summary.audio or tf.summary.image, among others. The first one is the most used, it displays the variation of a scalar metric over time, usually the loss or the learning rate. All these visualizations admit some dynamic interactions within the TensorBoard interface. Using TensorBoard within Keras high-level API it is even easier. It is just necessary to include the callback tensorboard function while training, specifying the log directory. 5https://github.com/tensorflow/tensorboard 4.4. IMAGE DATASETS 25 Figure 4.3: TensorBoard web server This function will record data for loss and accuracy metrics, as well as other metrics specified in the compile function. 4.4 Image datasets For correctly training and accurately evaluating a good Machine Learning model, a suitable selection or elaboration of the dataset is crucial. Once we have the dataset, it is divided into three disjoint subsets: training, test and validation sets. Training set is used for training the model, fitting the model’s parameters; test set is used once the model is trained, to provide an unbiased evaluation of this final model; validation set, also called development set or dev set, is used while training the model, to tune the hyperparameters and to prevent overfitting. As Deep Learning researchers have a strong interest in the image analysis field, during these years a lot of large and medium scale datasets have been created for training these models and we can find many of them available online. The most popular ones are explained below. 26 CHAPTER 4. FRAMEWORK ImageNet ImageNet6is probably the most well-known image dataset in the Deep Learning and machine learning industry. This image dataset is used mainly for classification purposes and the classes are organized based on the nouns in the WordNet [Miller, 1995] hierarchy. This image database was created in 2009 [Deng et al., 2009] by querying image search engines with WordNet’s synsets. They extracted a great number of candidate images for each synset and then filtered them by crowd-sourcing in Amazon Mechanical Turk (AMT)7. Since 2010, ImageNet also runs the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) annual visual challenge [Russakovsky et al., 2015]. Nowadays, this challenge is hosted in Kaggle8and it has three categories: image classification for 1000 classes, object detection (detect the bounding box where the object lays) for 200 classes and object detection from video for 30 classes. As of today, ImageNet is one of the largest image database available. It has a total of 14,197,122 annotated images of a wide variety of categories, with an average of over five hundred images per synset. For the person’s high-level category (the one we have interest in), it holds around 952,000 images and it is divided in 2,035 synsets with an average of 468 images per synset. Images are annotated with their classification and 1,034,908 of them also with their bounding boxes. MS COCO MS COCO9(MicroSoft Common Objects in COntext) dataset [Lin et al., 2014] is one of the most widely used large-scale datasets for image description generation and also for object detection, object segmentation and person keypoints detection. Currently, the MS COCO dataset consists of over 328,000 images of 91 basic object types in naturally occurring contexts. Each image is labeled with at least five captions describing it, which leads to a total of 2.5 million captions, and bounding boxes for each object category that appears in the picture. They also contain 250,000 people annotated with their keypoints. This dataset has also given rise to image challenges for object detection, keypoints detection and image captioning. The latter is already closed, but there is an open evaluation server and API [Chen et al., 2015] to compare to state-of-the-art methods using several performance metrics such as BLEU, ROUGE, METEOR and CIDEr. 6http://www.image-net.org/ 7A well-known crowd-sourcing online platform. https://www.mturk.com/ 8https://www.kaggle.com/ 9http://cocodataset.org/#home 4.4. IMAGE DATASETS 27 PASCAL PASCAL10 (Pattern Analysis Statistical Modeling and Computational Learning) Visual Object Classes project comprised challenges [Everingham et al., 2010] of image classification, detection and segmentation for eight consecutive years (2005-2012). Nowadays, the challenges are closed but it provides an image database and tools for obtaining these images and their annotations, as well as an evaluation server. The latest dataset (PASCAL 2012 challenge) contained 20 distinct classes and more than 11,000 annotated images. These images were collected from Flickr. These datasets have annotations for object classification, detection and segmentation, but a subset of the Pascal 2008 challenge dataset was used to create an image description database, called Pascal1K. It contains 1,000 images with objects of different classes and is annotated with five descriptions, generated by humans on Amazon Mechanical Turk crowd-sourcing platform. As it is a medium-scale database, it is generally used as a benchmark for evaluating image description models. Flickr8K and Flickr30K Flickr8K11 [Hodosh et al., 2013] dataset and its extension Flickr30K12 [Young et al., 2014], contain images from Flickr, each one annotated with five descriptions collected from the crowd-source platform Amazon Mechanical Turk (AMT). Flickr8K consists of approximately 8,000 images and Flickr30K of around 30,000 images. These pictures were collected by the University of Illinois by querying Flickr for specific objects and actions. SBU SBU Captioned Photo Dataset was specifically created for the Im2Text image caption generator [Ordonez et al., 2011]. It contains 1 million Flickr images labeled with their original captions generated by their users, collected by querying Flickr for specific objects and actions. It is a large-scale dataset but not very popular for training image description models as the labeled captions may contain subjective information or information not contained in the picture because they were generated by the Flickr user. CIFAR CIFAR-10 and CIFAR-100 (Canadian Institute for Advanced Research) [Krizhevsky, 2012] are image classification datasets that differ only in the number of classes they have (CIFAR10 has ten classes and CIFAR-100 one hundred classes, grouped into twenty superclasses). 10Download page: http://host.robots.ox.ac.uk/pascal/VOC/ 11Form for downloading the Filckr8K dataset:https://forms.illinois.edu/sec/1713398 12Form for downloading the Filckr30K dataset: https://forms.illinois.edu/sec/229675 28 CHAPTER 4. FRAMEWORK Figure 4.4: COCO caption dataset Both datasets consist of 60,000 32x32 RGB images, CIFAR-10 holds 6,000 in each class and CIFAR-100 holds 600. Keras provides built-in functions to load these datasets. LabelMe LabelMe13 is an image database and an online annotation tool [Russell et al., 2008] created in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). The dataset is dynamic, free to use and open to public contribution, and it consists of almost 200,000 images but not all of them are fully annotated. It has more than 62,000 annotated images and more than 658,000 labeled objects. The online annotation tool allows the user to draw polygons, query images, browse the database or download a subset. 4.4.1 Dataset election Having studied the characteristics of all these datasets, we decided to use the MS COCO dataset for training and testing our model, as it contains a wide number of images, it is one of the most commonly used for image description and it provides tools for downloading the dataset. They already provide a train/val/test split of the dataset, however, as this dataset was released for their annual challenges, the test set does not come labeled and we will not be able to use it for testing our model, so we will create our own split based on the training and validation sets. 13Download page: http://labelme.csail.mit.edu 4.4. IMAGE DATASETS 29 Figure 4.4 shows some examples of what can be found in COCO caption dataset. In particular, we use the Python COCO API to download the MS COCO dataset and to select a subset of it containing only images of people, to restrict to our objectives the images we provide to the model. In addition, we also use the people’s subset (baby, boy,girl,man and woman categories) of the CIFAR-100 dataset to fine-tune the image feature-extraction model we use, as explained in the next chapter. 36 CHAPTER 5. DEEP LEARNING APPROACH """-----Definition of VGG16 fine-tuned model-----""" def model_definition(): vgg16_model = VGG16(weights="imagenet", include_top=False, input_shape=(224,224,3)) # Replace top layer vgg16_output = vgg16_model.output x1 = Flatten()(vgg16_output) x2 = Dense(4096, activation=’relu’, name="fc1")(x1) x3 = Dense(4096, activation=’relu’, name="fc2")(x2) prediction = Dense(100, activation=’softmax’, name="predictions")(x3) # Create model model = Model(input=vgg16_model.input, output=prediction) # Set first layers non-trainable for layer in model.layers[:12]: layer.trainable = False model.compile(optimizer=SGD(lr=1e-3, decay=1e-6, momentum=0.9), loss=’categorical_crossentropy’, metrics=[’accuracy’]) return model Figure 5.4: Python code of VGG16 fine-tuned model definition. function with a batch size of 32 images and 30 epochs3, and save it in a hdf54file into memory using model.save(). 5.3 Data preparation Once we have our fine-tuned VGG16 model, we can start defining the models for generating image descriptions. As we explained in the first section of this chapter, we will use COCO dataset, partly or entirely depending on the model, for training and testing our models. Fortunately, COCO provides a Python API for loading into memory and parsing the dataset in an easier way. To use this API, the dataset files must be downloaded from their webpage in an specific directory structure. We have downloaded the 2014 dataset because it is the one that contains image descriptions as labels. After cloning the github project and running the Makefile, the API can be used by importing the pycocotools.coco 3An epoch is an entire iteration over the whole training dataset. The batch size is the number of training samples trained between parameters updates. 4File format to store large amounts of data. 5.3. DATA PREPARATION 37 module. More information on how to use this API can be found on their webpage5and their github project6. All COCO images and descriptions must be preprocessed in order to correctly use them as the inputs of our models. Even though two of our models work only with a subset of COCO, the other two work with the whole dataset, so both image preprocessing and descriptions preprocessing have been done in the whole dataset, leaving the subset extraction to a later step where we also perform the train/val/test split of the dataset as we explained in Section 4.4. In the following subsections we explain how we have carried out these two tasks. 5.3.1 Image preprocessing COCO images do not need too much preprocessing. However, the Convolutional Neural Network of our models, either the VGG16 network or the fine-tuned model, has already precomputed its weights and these weights will remain frozen when training our image descriptions generation models. Therefore, images features could be precomputed before training the models and saved to be used later as an input instead of the images themselves. This way we only calculate these features once instead of every time the training process passes through one of these images, saving computational time and memory space. This image preprocessing must be repeated for the VGG16 model and for our finetuned model, to extract the features obtained by both models and use one or another depending on which of the four models we are training. Figure 5.5 shows the features extraction function we have defined, where model refers to either VGG16() or to our finetuned model, features is the dictionary that will store the extracted image features and images ids are all COCO training image ids obtained with COCO API as follows: dataDir = ".." dataType = "train2014" annFile = ’{}/annotations/instances_{}.json’.format(dataDir,dataType) coco = pycocotools.coco.COCO(annFile) imgs_ids = coco.getImgIds() This process is also done with the validation COCO dataset, changing dataType to "val2014". To extract image features, last layer of the model must be removed using layers.pop() because this layer is the one that predicts which class the image belongs to and we only care about image features. We then load every image in the COCO dataset using COCO API and Keras load img() function. Image target size is 224x224 as this is the input size of VGG16 (and therefore also of our fine-tuned model). Once we have loaded the image into memory, it is converted to a numpy array and reshaped, using Keras functions from the keras.preprocessing.image module. Finally, features are extracted 5http://cocodataset.org/download 6https://github.com/cocodataset/cocoapi 38 CHAPTER 5. DEEP LEARNING APPROACH def features_extraction(model, features, directory, dataType, images_ids): # remove prediction layer from model model.layers.pop() model = Model(inputs=model.inputs, outputs=model.layers[-1].output) for id in images_ids: file = "0:0>12".format(id) path = "/images//COCO__.jpg".format(directory, dataType, dataType, file) img = load_img(path, target_size=(224, 224)) # img to 3D numpy array (height, weight, color channel) and reshape it img = img_to_array(img) img = img.reshape((1, img.shape[0], img.shape[1], img.shape[2])) # get features and store them in dict img = preprocess_input(img) feature = model.predict(img, verbose=1) features[id] = feature return features Figure 5.5: Image preprocessing: features extraction of COCO images. using the predict() function and then stored into the dictionary. After having extracted all training and validation images features, we save the dictionary containing them in a file, using Python pickle library7, for later use. 5.3.2 Text preprocessing The image descriptions preprocessing task that we have implemented performs various operations on the words to clean those descriptions and obtain a suitable vocabulary for learning, that should be representative and expressive enough. These operations are: changing uppercase letters to lowercase, removing punctuation marks, removing numbers, removing words of length one (as they are not expressive), removing words used less than five times in the whole dataset and transforming English contractions such as don’t to do not. We also add initial and final tokens (START and END) for delimiting the descriptions. Figure 5.6 shows the preprocessing function we have defined for cleaning COCO descriptions, where: word freq is a Python Counter8with the words’ frequencies and descriptions is a list containing all the descriptions of COCO training dataset and it is obtained using COCO API as follows: 7https://docs.python.org/3/library/pickle.html 8https://docs.python.org/3/library/collections.html#collections.Counter 5.4. DATASET SPLIT 39 def preprocess_desc(descriptions, word_freq): # translation table for removing punctuation table = str.maketrans(’’, ’’, string.punctuation) for desc_info in descriptions: desc = desc_info[’caption’] words = desc.split() # convert to lower case words = [w.lower() for w in words] # remove numbers words = [w for w in words if not w.isnumeric()] # transform contractions words = [contractions(w) for w in words] # remove punctuation from each word words = [w.translate(table) for w in words] # remove words length < 2 words = [w for w in words if len(w)>1] # remove words used < 5 times words = [w for w in word_freq.keys() if word_freq[w]>=5] # store in dictionary with START and END tokens desc_info[’caption’] = ’START ’ + ’ ’.join(words) + ’ END’ Figure 5.6: Text preprocessing: cleaning COCO descriptions. dataDir = ".." dataType = "train2014" annFile = ’/annotations/captions_.json’.format(dataDir,dataType) coco = COCO(annFile) anns_ids = coco.getAnnIds() descriptions = coco.loadAnns(anns_ids) This process is also repeated with the validation descriptions, as we have done with image preprocessing, changing dataType to "val2014". Each element of the descriptions list is a dictionary containing the image id the description belongs to, an id for the description and the description itself. For transforming contractions, we have defined a dictionary containing the transformations and contractions() returns this transformation if the word is in that dictionary. Once we have preprocessed all training and validation descriptions, we merge both lists into a single one and save it in a file using pickle library. 5.4 Dataset split After having preprocessed the entire COCO dataset, we now split it into train, validation and test sets (with a 70%-20%-10% division) and divide consequently the features dictionaries (VGG16 features and fine-tuned model features) obtained while preprocessing the 40 CHAPTER 5. DEEP LEARNING APPROACH # get ids of 35000 training imgs general and with people dataDir = ".." dataType = "train2014" annFile = ’/annotations/instances_.json’.format(dataDir,dataType) coco = COCO(annFile) cats_ids = coco.getCatIds(catNms=[’person’]) train_people_ids = coco.getImgIds(catIds=cats_ids) train_people_ids = random.sample(train_people_ids, 35000) train_ids = coco.getImgIds() train_imgs_ids = random.sample(train_ids, 35000) # get ids of 35000 validation imgs general and with people dataDir = ".." dataType = "val2014" annFile = ’/annotations/instances_.json’.format(dataDir,dataType) coco = COCO(annFile) cats_ids = coco.getCatIds(catNms=[’person’]) val_people_ids = coco.getImgIds(catIds=cats_ids) val_people_ids = random.sample(val_people_ids, 15000) val_ids = coco.getImgIds() val_imgs_ids = random.sample(val_ids, 10000) # split val people in val and test val_people_ids, test_ids = train_test_split(val_people_ids, test_size=5000) Figure 5.7: Test split of COCO dataset. images and the descriptions list obtained while preprocessing the descriptions. As our goal is to evaluate how well our models describe images containing people, the test set must only include this kind of images. On the other hand, the train and validation sets, may or may not include other kind of images, depending on the model that we are training. Therefore, we will have two different splits for these two sets, one of them of the subset of images containing people (excluding the ones in the test set) and the other one of the whole dataset without the test set. Training our models with the whole COCO dataset (more than 300,000 images) was impossible due to the characteristics of the server we have used and to our time limitation. In order to reduce time and memory consumption, we decided to restrict our training process to only a subset of the COCO dataset: 35,000 images for the training set, 10,000 images for validation and 5,000 images for the test set (70%-20%-10%). To obtain the training set we have randomly selected, using random.sample() Python function, 35,000 images of the whole COCO training set and 35,000 images of all the COCO training images that contain people. Validation and test sets have been extracted both 5.5. TRAINING PROCESS 41 from COCO validation set, randomly selecting 15,000 COCO validation images containing people (5,000 for the test set and the remaining 10,000 for the validation set) and 10,000 COCO validation images of any kind. This process is shown in Figure 5.7. We have also ensure that the general training and validation splits are representative enough by verifying that the random subsets have enough images that contain people (30%-60%). Once we have performed the dataset splits, we divide the features dictionaries and the descriptions list accordingly to the different splits and we save these divisions in memory using pickle library. 5.5 Training process The first step in the training process of the four models is to load the files obtained with the image and text preprocessing and divided when making the dataset split. Following the enumeration we have defined in Section 5.1, Model 1 and Model 3 load image features obtained with VGG16 and Model 2 and Model 4 load the ones obtained with the finetuned VGG16 model. Subsequently, Model 1 and Model 2 load the train and validation split of the whole dataset and Model 3 and Model 4 the one of the subset of images that contain people. The remaining steps in the training process are the same for the four models, with the only differences being the inputs. After loading the images features and descriptions, we create and fit a Tokenizer over all training and validation descriptions, using Keras Tokenizer() class. A Tokenizer learns from a list of texts how to map the bag of words contained in that texts to integer values and allowing to see each description as an integer sequence. The next step is to define the model that we are going to train. The model definition in Python using Keras follows the architecture that we have defined in Figure 5.2 and it is shown in Figure 5.8. The hyperparameters are extracted from Tanti et al. observations [Tanti et al., 2017b]. vocab size,max length and emb matrix arguments are the size of the whole bag of words contained in the descriptions, the length of the largest description and the GloVe word embedding weight matrix respectively. The function starts with the definition of the Keras input tensor for the image features using Input() and the input shape is 4096 as it is the shape of the features vector extracted for each image. This input is passed to a Dense layer with a ReLu activation function and L2 regularization to obtain a 128 element representation. Regularization techniques are introduced in Deep Learning training processes to prevent or reduce overfitting (good learning of the traning set but too specific, leading to no generalization power). In particular, L2 regularization adds an extra term to the cost function proportional to the sum of squares of all the weights. We then define the language model with a Keras Input() tensor of length max length and this input is passed to a word embedding layer with GloVe pre-trained weights to map the sparse word vectors to dense ones, as we explained in Section 5.1.2. After the Embedding layer we perform a 0.5 dropout [Srivastava et al., 2014], 42 CHAPTER 5. DEEP LEARNING APPROACH def model_definition(vocab_size, max_length, emb_matrix): # feature extractor model img_inputs = Input(shape=(4096,)) fe1 = Dense(128, kernel_regularizer=l2(1e-8), activation=’relu’, name="features")(img_inputs) # description model desc_inputs = Input(shape=(max_length,)) de1 = Embedding(vocab_size, 100, weights=[emb_matrix], trainable=False)(desc_inputs) de2 = Dropout(0.5, name="text_dropout")(de1) de3 = LSTM(128, name="LSTM_layer")(de2) # merge model merge_inputs = [fe1,de3] me1 = add(merge_inputs) me2 = Dense(128, activation=’relu’, name="decoder")(me1) outputs = Dense(vocab_size, kernel_regularizer=l2(1e-8), activation=’softmax’, name="predictions")(me2) model = Model(inputs=[img_inputs, desc_inputs], outputs=outputs) model.compile(loss=’categorical_crossentropy’, optimizer=’adam’, metrics=[’accuracy’]) model.summary() return model Figure 5.8: Image description generation model definition with Keras. a regularization technique used to prevent overfitting that consists in ignoring while training some neurons selected randomly. This is followed by a Keras LSTM layer with 128 memory units. Next we merge both the output of the LSTM layer and the output of the Dense layer and apply a Dense layer with ReLU activation function and a softmax Dense layer, with L2 regularization, over the vocabulary size to make the prediction of the next word. This model is trained with a categorical cross entropy loss (as the output of the network are the probabilities of the words in the vocabulary to be the next word in the description) and Adam optimizer (see Section 5.5.1) with the default values, as Tanti et al. suggest [Tanti et al., 2017b]. Finally, the last step is to fit the model with the training and validation data that we have loaded. As COCO is a very big dataset and to ease memory usage, we use a generator to produce training and validation data and fit the model with the fit generator function. To control the training process, we use the fit generator function with a ModelCheckpoint callback, to save the model after each epoch monitoring the loss on the validation set, and a TensorBoard callback, to save tensorboard logs for visualization (as we explained in Section 4.3). 5.5. TRAINING PROCESS 43 Figure 5.9: Optimization algorithms steps reaching local optimum. We have trained the four models 35 epochs in the server with the GPU. The whole process (VGG16 fine-tune, data preparation and training of the four models) lasted around eight days. 5.5.1 Optimizers As we explained in Section 3.4, Deep Learning models learn by trying to find the optimal parameters that minimize the cost function and the gradient descent method is an optimization algorithm to approximate a local optimum by computing the gradient of the function. However, the cost function depends on all training samples, and therefore so does the gradient, which makes it too expensive to compute for Deep Learning models that are typically trained on a big dataset. For this reason, as the cost function is usually a sum over the training set, an incremental approach for calculating the gradient called Stochastic Gradient Descent (SGD) was developed. It calculates the gradient based only on a small subset of the training examples to speed up this calculation. This usually involves taking more iterations to approximate the minimum, but faster. In SGD, weights are updated using the same learning rate for all weights and all iterations, making it difficult or slow to converge to the minimum if the learning rate is too big or to small. Many variants have been proposed and developed in order to update this SGD algorithm to try to reach faster the local minimum. Some of this variants are AdaGrad [Duchi et al., 2011], Adam [Kingma and Ba, 2014] and RMSProp. AdaGrad maintains a per-parameter learning rate instead of having the 44 CHAPTER 5. DEEP LEARNING APPROACH same learning rate for all the parameters, updating it based on the sum of previous gradient values. So does RMSProp, that also has a learning rate for each parameter but these learning rates are updated based on the average of recent gradient calculations, that is, based on how quickly the gradient is changing. Adam is an improvement of RMSProp that updates the learning rates based on the average but also on the gradients variance, this is the one we use for training our model. Figure 5.9, obtained from Google Images, shows how quickly these algorithms reduce the cost function. As mentioned before, our models have been trained using the Adam optimizer. Chapter 6 Evaluation and results Once we have our four models defined and trained, we can evaluate them on the test set and we can use them to generate descriptions of new unseen images. In this chapter, we explain how we have tested the models, we define the metrics that we have used and we show the results obtained, including some examples of images and their generated descriptions. In Section 5.4 we explained that, as our objective is to determine how well our models perform on describing images that contain people, the division of the dataset was made so that the test set has only this kind of images. Therefore, the results of the evaluation process and the examples that we show in this chapter refer all to images containing people. 6.1 TensorBoard visualization As described in Section 5.1, the four models that we have developed follow the same architecture but differ on the techniques and datasets used. The characteristics of these four models are the following: •Model 1: the most general one. Uses a pre-trained VGG-16 CNN and it is trained on MS COCO dataset. •Model 2: uses a fine-tuned version of VGG-16 and it is trained on MS COCO dataset. •Model 3: uses a pre-trained VGG-16 CNN and it is trained on a subset of MS COCO dataset of images containing people. •Model 4: uses a fine-tuned version of VGG-16 and it is trained on a subset of MS COCO dataset of images containing people. 45 52 CHAPTER 6. EVALUATION AND RESULTS Model 1: person in the snow holding snowboard down snow Model 3: man on skis in the snow Figure 6.8: Good descriptions. Model 1: woman on beach with surfboard Model 3: two woman riding surfboard on the beach Figure 6.9: Good descriptions. 6.3. QUALITATIVE ANALYSIS 53 Model 1: man in baseball uniform is holding bat Model 3: baseball player is swinging bat at baseball game Figure 6.10: Good descriptions. Model 1: bird flying on beach Model 3: man is riding surfboard on the beach Figure 6.11: Different perspectives. 54 CHAPTER 6. EVALUATION AND RESULTS Model 1: pizza with cheese and cheese on it Model 3: person is sitting at table with pizza Figure 6.12: Different perspectives. Model 1: plate of food with meat and vegetables on it Model 3: person is sitting on table with some food Figure 6.13: Different perspectives. 6.3. QUALITATIVE ANALYSIS 55 Model 1: person in the snow on skis on the snow Model 3: man riding snowboard down snow covered slope Figure 6.14: Bad descriptions. Model 1: man holding hot dog in front of pizza Model 3: man is sitting at table with pizza Figure 6.15: Bad descriptions. 56 CHAPTER 6. EVALUATION AND RESULTS Chapter 7 Conclusions and future work 7.1 Conclusions We conclude this project having accomplished the main objectives established at the beginning. We have studied and learned different Deep Learning techniques and tools, that are nowadays very important in the Artificial Intelligence field. We have also reviewed all the bibliography regarding the image description generation problem. We have carried out this project in a server with a GPU provided by GAIA research group, as training Deep Learning models needs a lot of computing capacity. We have used Keras on top of Tensorflow as the Deep Learning framework and TensorBoard for generating plots of the training process. We have also analyzed the different available and well-known image datasets in order to select the ones that were more suitable for our project. Finally, we have achieved to develop four different Deep Learning models to automatically generate descriptions of images containing people. Our four models were all built using state-of-the-art Deep Learning techniques for Computer Vision and for Natural Language Generation: Convolutional Neural Networks for extracting the image features and Recurrent Neural Networks for generating the sentence, as well as Feedforward Neural Networks for building the models’ outputs. These models were successfully trained and tested on a big and popular image dataset. To evaluate our models we have analyzed the TensorBoard plots generated from the training data, we have performed a quantitative analysis using the most popular metrics and a qualitative analysis generating descriptions for unseen images in order to visualize and confirm the results obtained from the quantitative analysis. These evaluation analyses lead all to the same conclusions: we obtained desirable results from the evaluation process, similar to state-of-the-art results, for two of the four models developed (Model 1 and Model 3). However, fine-tuning was not a good practice, as Model 2 and Model 4 (the ones that fine-tuned the CNN) have poor evaluation results, 57 58 CHAPTER 7. CONCLUSIONS AND FUTURE WORK maybe because of the differences between the datasets used for fine-tuning and training. Nevertheless, Model 3 have slightly better results than Model 1 so we can confirm that our thoughts were in the right path at the beginning of the project: images containing people are described differently and so it is a good practice to restrict the training process to this kind of images. 7.2 Future work Considering concluded the main objectives of this project, there are many directions in which future work could go in order to broaden it. First of all, data quantity and quality are essential for a good training, and hence the model could be more accurate and have better evaluation results with more, and more representative, examples in the dataset and labeled more precisely. Another option is to improve our hardware capacity by using more GPUs, a better one or by another way of obtaining more computational power in order to being able to train the models more rapidly. In addition to all of these changes to accomplish better results with our models, another direction we could take in the future is to broaden the scope of the model’s objective. We could include more complete image descriptions depicting more complex characteristics of the person in the image, such as the person’s feelings (e.g. if it is smiling or sad), or other characteristics that are not explicitly shown in the picture but that can be derived from it, not limiting ourselves to describing what it is strictly in the picture. Moreover, the model could be extended to the description of, not only people, but other objects appearing in the picture, as well as the spatial relationship between those objects and the people (e.g. the man behind the window,the woman next to the blue chair). There are also some emerging tasks that combine both computer vision and natural language fields, trying to go beyond what it is strictly depicted in the image, that could be interesting to study in a future work of this project; for example, the task of VQA (Visual Question Answering) [Wu et al., 2017a], that tries to find an answer to a given question about a given image, or visual storytelling [Huang et al., 2016, Mostafazadeh et al., 2017], that generates a narrative description from the input image by making subjective assumptions of what it is happening. Another interesting possibility is to implement a machine learning model, using stateof-the-art machine learning algorithms such as xgboost, in order to analyse the differences between both deep learning and machine learning models (computing time, dataset needs, accuracy, etc). Finally, although Convolutional Neural Networks are the state-of-the-art, they have some limitations that decrease the level of accuracy of the model. In the last months, a new kind of neural networks trying to solve these limitations, called Capsule Neural Networks, have arisen. It would be a good future direction of this project to implement 7.2. FUTURE WORK 59 the solution using this kind of network and to compare it to the convolutional one to see if it yields better results. 7.2.1 CNN limitations Convolutional Neural Networks are until now the state-of-the-art approach in the image analysis field, but they have some limitations when it comes to spatial relationships. Convolutional Neural Networks became a major advantage compared to traditional feedforward networks in the image field, as they allowed a 2-D matrix as the input of the model in contrast with the need of flattening the pixel matrix into a vector, which, along with the convolution and pooling operations, permitted the model to be invariance to translations. As we have already discussed during this project, the job of the convolutional layers in a CNN is to detect important features in the image, with layers closer to the input detecting simple features and deeper layers close to the output combining them to detect more complex features. On the other hand, pooling layers, being max pooling the most widely used, help reducing the image dimensionality, and therefore computational time, and summarizing the important information present in the image, as well as creating spatial invariance. However, this invariance created by the max pooling layers loses the existing spatial relationships between all these features. It does not take into account how the different features are related to each other, since max pooling loses their precise locations. This makes the model output a false positive when the image has the components of an object but not in the correct order. For example, considering it is a face when it has two eyes, a nose and a mouth, but the mouth is where an eye should be and vice versa. Furthermore, CNNs do not take into account the spatial characteristics of each feature. That is, they do not recognise an object they have already seen, shown now in a different orientation. Continuing with the example above, an image with a face turned upside down is not detected as a face by the CNN. To combat this, convolutional models are trained by giving images of different possible angles explicitly to the network. Nevertheless, the solution is not to remove the max pooling layers from the model, as we really need to introduce some kind of invariance; otherwise, the model will only recognise images very similar to the ones in the training set. What a better model needs is to adjust this invariance with equivariance, that is, understanding rotation and proportion changes and adapting accordingly. 7.2.2 Capsule neural networks Geoffrey E. Hinton, known as the father of Deep Learning, has been discussing about convolutional networks’ limitations and the need to suggest new models to cover them, for 60 CHAPTER 7. CONCLUSIONS AND FUTURE WORK Figure 7.1: Capsule neural network structure (CapsNet). a long time now. In 2011 he published a paper [Hinton et al., 2011] regarding a proposal of including more complex spatial characteristics to neural networks, but it has not been until recently that he has finally published two papers [Hinton et al., 2018, Sabour et al., 2017] explaining his new sophisticated neural network model, Capsule Networks. In the paper they have published [Sabour et al., 2017], they propose CapsNet, a capsule neural network model for the MNIST1dataset, reporting better results than CNNs and without the need of data augmentation. This model is represented in Figure 7.1. Hinton believes that it is necessary to preserve pose’s (translation and rotation) hierarchy between features. This way it is easier for a model to recognise another perspective of something it has already seen. These neural networks try to overcome those Convolutional Networks’ limitations by explicitly taking into account those spatial relationships between features, whereas CNNs do not have this 3-D space understanding. This is achieved by replacing the scalar outputs of traditional neurons with vector outputs encoding the features’ locations and changing max-pooling with a dynamic routing algorithm. Thus, an object presence is derived from not only the presence of its constituent parts, but also from them being at the right locations. According to Hinton, A capsule is a group of neurons whose outputs represent different properties of the same entity. They are nested layers within a layer, each one focusing on detecting a particular feature in the image and outputting a vector representing the feature’s existence and its pose properties. The length of the vector represents the probability of the existence of the feature and the spatial properties are encoded in the vector’s direction. This way, when the detected feature moves or changes its spatial state, the length of the vector does not change (the probability remains the same) but it changes its orientation. In these neural networks, capsules in a lower layer decide dynamically how to send its output vector to the next layer’s capsules. Each capsule in the lower layer computes, for each possible parent, a prediction vector of the pose of a higher-level capsule feature 1Dataset of handwritten digits. 7.2. FUTURE WORK 61 (what the higher capsule would see), by multiplying its output by a weight matrix. When several capsules in one layer agree on what they may have detected, they activate the corresponding capsule at the next layer. Hinton calls this method the routing-by-agreement algorithm, and it substitutes the max-pooling algorithm. 68 CHAPTER 8. CONCLUSIONES Y TRABAJO A FUTURO Bibliography [DBL, 2015] (2015). IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. IEEE Computer Society. [Abadi et al., 2016] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P. A., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zhang, X. (2016). Tensorflow: A system for large-scale machine learning. CoRR, abs/1605.08695. [Altman, 1992] Altman, N. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175–185. [Baik and Bala, 2004] Baik, S. and Bala, J. W. (2004). A decision tree algorithm for distributed data mining: Towards network intrusion detection. In Lagan`a, A., Gavrilova, M. L., Kumar, V., Mun, Y., Tan, C. J. K., and Gervasi, O., editors, Computational Science and Its Applications - ICCSA 2004, International Conference, Assisi, Italy, May 14-17, 2004, Proceedings, Part IV, volume 3046 of Lecture Notes in Computer Science, pages 206–212. Springer. [Bay et al., 2006] Bay, H., Tuytelaars, T., and Gool, L. J. V. (2006). SURF: speeded up robust features. In Leonardis, A., Bischof, H., and Pinz, A., editors, Computer Vision - ECCV 2006, 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings, Part I, volume 3951 of Lecture Notes in Computer Science, pages 404–417. Springer. [Bengio et al., 1994] Bengio, Y., Simard, P. Y., and Frasconi, P. (1994). Learning longterm dependencies with gradient descent is difficult. IEEE Trans. Neural Networks, 5(2):157–166. [Bernardi et al., 2016] Bernardi, R., C¸akici, R., Elliott, D., Erdem, A., Erdem, E., IkizlerCinbis, N., Keller, F., Muscat, A., and Plank, B. (2016). Automatic description generation from images: A survey of models, datasets, and evaluation measures. J. Artif. Intell. Res., 55:409–442. 69 70 BIBLIOGRAPHY [Bridge et al., 2014] Bridge, J. P., Holden, S. B., and Paulson, L. C. (2014). Machine learning for first-order theorem proving - learning to select a good heuristic. J. Autom. Reasoning, 53(2):141–172. [Canny, 1986] Canny, J. F. (1986). A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 8(6):679–698. [Chen et al., 2015] Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Doll´ar, P., and Zitnick, C. L. (2015). Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. [Chen and Zitnick, 2015] Chen, X. and Zitnick, C. L. (2015). Mind’s eye: A recurrent visual representation for image caption generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 2422–2431. [Cho et al., 2014] Cho, K., van Merrienboer, B., G¨ul¸cehre, C¸ ., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In [Moschitti et al., 2014], pages 1724–1734. [Chung et al., 2014] Chung, J., G¨ul¸cehre, C¸ ., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. [Collobert et al., 2011] Collobert, R., Kavukcuoglu, K., and Farabet, C. (2011). Torch7: A matlab-like environment for machine learning. [Cortes and Vapnik, 1995] Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3):273–297. [Dean et al., 2012] Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. (2012). Large scale distributed deep networks. In NIPS. [Deng et al., 2009] Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society. [Denkowski and Lavie, 2014] Denkowski, M. J. and Lavie, A. (2014). Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 2627, 2014, Baltimore, Maryland, USA, pages 376–380. The Association for Computer Linguistics. [Donahue et al., 2015] Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., and Saenko, K. (2015). Long-term recurrent convolutional networks for visual recognition and description. In [DBL, 2015], pages 2625–2634. BIBLIOGRAPHY 71 [Duchi et al., 2011] Duchi, J. C., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. [Everingham et al., 2010] Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–338. [Goldberg et al., 1994] Goldberg, E., Driedger, N., and Kittredge, R. I. (1994). Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45–53. [Graves et al., 2013] Graves, A., Mohamed, A., and Hinton, G. E. (2013). Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pages 6645–6649. IEEE. [Harris and Stephens, 1988] Harris, C. G. and Stephens, M. (1988). A combined corner and edge detector. In Taylor, C. J., editor, Proceedings of the Alvey Vision Conference, AVC 1988, Manchester, UK, September, 1988, pages 1–6. Alvey Vision Club. [Hendricks et al., 2016] Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R. J., Saenko, K., and Darrell, T. (2016). Deep compositional captioning: Describing novel object categories without paired training data. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1–10. [Hinton et al., 2011] Hinton, G. E., Krizhevsky, A., and Wang, S. D. (2011). Transforming auto-encoders. In Honkela, T., Duch, W., Girolami, M. A., and Kaski, S., editors, Artificial Neural Networks and Machine Learning - ICANN 2011 - 21st International Conference on Artificial Neural Networks, Espoo, Finland, June 14-17, 2011, Proceedings, Part I, volume 6791 of Lecture Notes in Computer Science, pages 44–51. Springer. [Hinton et al., 2018] Hinton, G. E., Sabour, S., and Frosst, N. (2018). Matrix capsules with EM routing. In International Conference on Learning Representations. [Ho, 1995] Ho, T. K. (1995). Random decision forests. In Third International Conference on Document Analysis and Recognition, ICDAR 1995, August 14 - 15, 1995, Montreal, Canada. Volume I, pages 278–282. IEEE Computer Society. [Hochreiter and Schmidhuber, 1997] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735–1780. [Hodosh et al., 2013] Hodosh, M., Young, P., and Hockenmaier, J. (2013). Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res., 47:853–899. [Hotelling, 1933] Hotelling, H. (1933). Analysis of a complex of statistical variables with principal components. Journal of Educational Psychology, 24:417–441. 72 BIBLIOGRAPHY [Huang et al., 2016] Huang, T. K., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R. B., He, X., Kohli, P., Batra, D., Zitnick, C. L., Parikh, D., Vanderwende, L., Galley, M., and Mitchell, M. (2016). Visual storytelling. In Knight, K., Nenkova, A., and Rambow, O., editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1233–1239. The Association for Computational Linguistics. [Iordanskaja et al., 1992] Iordanskaja, L., Kim, M., Kittredge, R. I., Lavoie, B., and Polgu`ere, A. (1992). Generation of extended bilingual statistical reports. In 14th International Conference on Computational Linguistics, COLING 1992, Nantes, France, August 23-28, 1992, pages 1019–1023. [Jia et al., 2014] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. [Karpathy and Li, 2015] Karpathy, A. and Li, F. (2015). Deep visual-semantic alignments for generating image descriptions. In [DBL, 2015], pages 3128–3137. [Kingma and Ba, 2014] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. CoRR, abs/1412.6980. [Kiros et al., 2014a] Kiros, R., Salakhutdinov, R., and Zemel, R. S. (2014a). Multimodal neural language models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pages 595–603. [Kiros et al., 2014b] Kiros, R., Salakhutdinov, R., and Zemel, R. S. (2014b). Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539. [Krizhevsky, 2012] Krizhevsky, A. (2012). Learning multiple layers of features from tiny images. [Krizhevsky et al., 2012] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Bartlett, P. L., Pereira, F. C. N., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pages 1106–1114. [Kulkarni et al., 2013] Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., and Berg, T. L. (2013). Babytalk: Understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 35(12):2891–2903. [Kuznetsova et al., 2012] Kuznetsova, P., Ordonez, V., Berg, A. C., Berg, T. L., and Choi, Y. (2012). Collective generation of natural image descriptions. In The 50th BIBLIOGRAPHY 73 Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 1: Long Papers, pages 359– 368. The Association for Computer Linguistics. [LeCun et al., 2015] LeCun, Y., Bengio, Y., and Hinton, G. E. (2015). Deep learning. Nature, 521(7553):436–444. [Li et al., 2011] Li, S., Kulkarni, G., Berg, T. L., Berg, A. C., and Choi, Y. (2011). Composing simple image descriptions using web-scale n-grams. In Goldwater, S. and Manning, C. D., editors, Proceedings of the Fifteenth Conference on Computational Natural Language Learning, CoNLL 2011, Portland, Oregon, USA, June 23-24, 2011, pages 220–228. ACL. [Lin, 2004] Lin, C.-Y. (2004). Rouge: a package for automatic evaluation of summaries. [Lin et al., 2014] Lin, T., Maire, M., Belongie, S. J., Bourdev, L. D., Girshick, R. B., Hays, J., Perona, P., Ramanan, D., Doll´ar, P., and Zitnick, C. L. (2014). Microsoft COCO: common objects in context. CoRR, abs/1405.0312. [Lowe, 1999] Lowe, D. G. (1999). Object recognition from local scale-invariant features. In ICCV, pages 1150–1157. [Mao et al., 2014] Mao, J., Xu, W., Yang, Y., Wang, J., and Yuille, A. L. (2014). Deep captioning with multimodal recurrent neural networks (m-rnn). CoRR, abs/1412.6632. [Mason and Charniak, 2014] Mason, R. and Charniak, E. (2014). Nonparametric method for data-driven image captioning. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 592–598. The Association for Computer Linguistics. [Miller, 1995] Miller, G. A. (1995). Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. [Mitchell et al., 2012] Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A. C., Berg, T. L., and III, H. D. (2012). Midge: Generating image descriptions from computer vision detections. In Daelemans, W., Lapata, M., and M`arquez, L., editors, EACL 2012, 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012, pages 747–756. The Association for Computer Linguistics. [Moschitti et al., 2014] Moschitti, A., Pang, B., and Daelemans, W., editors (2014). Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL. [Mostafazadeh et al., 2017] Mostafazadeh, N., Brockett, C., Dolan, B., Galley, M., Gao, J., Spithourakis, G. P., and Vanderwende, L. (2017). Image-grounded conversations: 74 BIBLIOGRAPHY Multimodal context for natural question and response generation. In Kondrak, G. and Watanabe, T., editors, Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers, pages 462–472. Asian Federation of Natural Language Processing. [Mozer, 1989] Mozer, M. C. (1989). A focused backpropagation algorithm for temporal pattern recognition. Complex Systems, 3(4). [Ordonez et al., 2011] Ordonez, V., Kulkarni, G., and Berg, T. L. (2011). Im2text: Describing images using 1 million captioned photographs. In Shawe-Taylor, J., Zemel, R. S., Bartlett, P. L., Pereira, F. C. N., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain., pages 1143–1151. [Pang et al., 2002] Pang, B., Lee, L., and Vaithyanathan, S. (2002). Thumbs up?: Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pages 79–86, Stroudsburg, PA, USA. Association for Computational Linguistics. [Papineni et al., 2002] Papineni, K., Roukos, S., Ward, T., and Zhu, W. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA., pages 311–318. ACL. [Pennington et al., 2014] Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In [Moschitti et al., 2014], pages 1532–1543. [Reiter and Dale, 1997] Reiter, E. and Dale, R. (1997). Building applied natural language generation systems. Natural Language Engineering, 3(1):57–87. [Russakovsky et al., 2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Li, F. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252. [Russell et al., 2008] Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1-3):157–173. [Sabour et al., 2017] Sabour, S., Frosst, N., and Hinton, G. E. (2017). Dynamic routing between capsules. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R., editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 3859–3869. BIBLIOGRAPHY 75 [Sarikaya et al., 2014] Sarikaya, R., Hinton, G. E., and Deoras, A. (2014). Application of deep belief networks for natural language understanding. IEEE/ACM Trans. Audio, Speech & Language Processing, 22(4):778–784. [Seide and Agarwal, 2016] Seide, F. and Agarwal, A. (2016). Cntk: Microsoft’s opensource deep-learning toolkit. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 2135–2135, New York, NY, USA. ACM. [Srivastava et al., 2014] Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. [Tanti et al., 2017a] Tanti, M., Gatt, A., and Camilleri, K. P. (2017a). What is the role of recurrent neural networks (rnns) in an image caption generator? In Proceedings of the 10th International Conference on Natural Language Generation, INLG 2017, Santiago de Compostela, Spain, September 4-7, 2017, pages 51–60. [Tanti et al., 2017b] Tanti, M., Gatt, A., and Camilleri, K. P. (2017b). Where to put the image in an image caption generator. CoRR, abs/1703.09137. [Theano Development Team, 2016] Theano Development Team (2016). Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688. [Tran et al., 2016] Tran, K., He, X., Zhang, L., and Sun, J. (2016). Rich image captioning in the wild. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2016, Las Vegas, NV, USA, June 26 - July 1, 2016, pages 434–441. IEEE Computer Society. [Vedantam et al., 2014] Vedantam, R., Zitnick, C. L., and Parikh, D. (2014). Cider: Consensus-based image description evaluation. CoRR, abs/1411.5726. [Vinyals et al., 2015] Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015). Show and tell: A neural image caption generator. In [DBL, 2015], pages 3156–3164. [Wernick et al., 2010] Wernick, M. N., Yang, Y., Brankov, J. G., Yourganov, G., and Strother, S. C. (2010). Machine learning in medical imaging. IEEE Signal Processing Magazine, 27(4):25–38. [Wu et al., 2017a] Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A. R., and van den Hengel, A. (2017a). Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21–40. [Wu et al., 2017b] Wu, S., Wieland, J., Farivar, O., and Schiller, J. (2017b). Automatic alt-text: Computer-generated image descriptions for blind users on a social network service. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’17, pages 1180–1192, New York, NY, USA. ACM. 76 BIBLIOGRAPHY [Wu et al., 2016] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., and Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. [Yang et al., 2011] Yang, Y., Teo, C. L., III, H. D., and Aloimonos, Y. (2011). Corpusguided sentence generation of natural images. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 444–454. ACL. [Young et al., 2014] Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67–78.