Deep Learning in Image Cytometry: A Review

Deep Learning in Image Cytometry: A Review Anindya Gupta, 1 Philip J. Harrison, 2 Håkan Wieslander, 1 Nicolas Pielawski, 1 Kimmo Kartasalo, 3,4 Gabriele Partel, 1 Leslie Solorzano, 1 Amit Suveer, 1 Anna H. Klemm, 1,5 Ola Spjuth, 2 Ida-Maria Sintorn, 1 Carolina Wählby 1,5 * Abstract Artificial intelligence, deep convolutional neural networks, and deep learning are all niche terms that are increasingly appearing in scientific presentations as well as in the general media. In this review, we focus on deep learning and how it is applied to microscopy image data of cells and tissue samples. Starting with an analogy to neuroscience, we aim to give the reader an overview of the key concepts of neural networks, and an understanding of how deep learning differs from more classical approaches for extracting information from image data. We aim to increase the understanding of these methods, while highlighting considerations regarding input data requirements, computational resources, challenges, and limitations. We do not provide a full manual for applying these methods to your own data, but rather review previously published articles on deep learning in image cytometry, and guide the readers toward further reading on specific networks and methods, including new methods not yet applied to cytometry data. © 2018 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of International Society for Advancement of Cytometry. Key terms biomedical image analysis; cell analysis; convolutional neural networks; deep learning; image cytometry; microscopy; machine learning AUTOMATION of microscopy, including sample handling and microscope control, enables rapid collection of digital image data from cell samples, tissue slides, and cell cultures grown in multi-well plates, transforming imaging cytometry into one of the most data-rich scientific disciplines. The first approaches to automated analysis of microscopy data appeared already in the 1950s, and a wealth of methods for finding cells and subcellular regions, and designing features that describe phenotypic variations in response to disease or potential drugs, have been developed over the years (1,2). These approaches, often combined with conventional machine learning methods, have been successfully used for many complex biological datasets (3,4). However, task-specific algorithm optimization and feature engineering is a challenging and time-consuming task, and often insufficient for dealing with global and local contextual variations. Thanks to increased computing power and large amounts of annotated images of natural scenes, methods based on the ideas about neural network and deep learning, that have been around for a long time, are finally working in practice and we now see the fast emergence of approaches to image analysis where the computer learns the task at hand from examples and automatically exploits the input images for measurements or decisions. A comparison between conventional and deep learning workflows is illustrated in Figure 1. Deep learning relates to fundamental concepts in neuroscience. A neuron is an electrically excitable cell that receives signals from other neurons, processes the received information, and transmits electrochemical signals to other neurons (5). The input signal to a given neuron needs to exceed a certain threshold for the 1 Centre for Image Analysis, Uppsala University, Uppsala, 75124, Sweden 2 Department of Pharmaceutical Biosciences, Uppsala University, Uppsala, 75124, Sweden 3 Faculty of Medicine and Life Sciences, University of Tampere, Tampere, 33014, Finland 4 Faculty of Biomedical Sciences and Engineering, Tampere University of Technology, Tampere, 33720, Finland 5 BioImage Informatics Facility of SciLifeLab, Uppsala, 75124, Sweden Received 5 October 2018; Revised 7 November 2018; Accepted 29 November 2018 Grant sponsor: European research council, Grant numberERC-2015-CoG 683810; Grant sponsor: Stiftelsen för Strategisk Forskning, Grant numberBD15-0008SB16-0046; Grant sponsor: Vetenskapsrådet, Grant number2014-6075 Additional Supporting Information may be found in the online version of this article. * Correspondence to: Carolina Wählby, Centre for Image Analysis, Uppsala University, Uppsala 75124, Sweden. Email: [email protected] Published online 19 December 2018 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/cyto.a.23701 © 2018 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of International Society for Advancement of Cytometry. This is an open access article under the terms of the Creative Commons AttributionNonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes. Cytometry Part A 95A: 366–380, 2019 REVIEW ARTICLE neuron to be activated and further transmit a signal. The neurons are interconnected and form a network that collectively steers brain mechanisms (6). Inspired by our brain, an artificial neural network is an abstracted interconnected network, consisting of neurons (perceptrons) grouped into layers. It consists of an input layer aggregating the input signals from other connected neurons, one or many hidden layers of thresholds or weights, and an output layer for predictions. Each neuron takes the input from the neurons of the previous layer using various weights (determined by training) and computes an activation function (e.g., sigmoid function) to stimulate a non-linear behavior, which is relayed onto the next layer of neurons. A deep neural network (DNN) is formed by cascading several such neurons in multiple layers to form a richer hierarchical network commonly known as a multilayer perceptron (MLP), where all neurons in the previous layer are densely (or fully) connected to the neurons of succeeding layers (7–10). The neurons within the layers may also be connected to themselves to enable the solving of more complex problems. However, they are restricted to input data structured in vector form, which limits their suitability to images. To overcome this, convolutional neural networks (CNNs) were developed (11), wherein 2D filters (or convolutional kernels) are used to process local image information in matrix form. From a biological perspective, a CNN approximately emulates the primate brain’s visual system (12–14), which employs a combination of convolutional and pooling layers (see section on “Overview of Key Concepts”below) before the dense layers to progressively encode richer representations in an image (15). Filter weights in “deep”CNNs are updated iteratively using annotated image examples. Hereafter we often refer to such methods as “deep learning”for brevity, though deep learning as such is a broader concept than CNNs. Deep learning has several advantages over conventional methods. For instance, these techniques directly learn discriminative representations (or features) from image examples, and effectively leverage the feature interaction and hierarchy within the data, resulting in a simplified featureextraction and selection process. Additionally, the performance of a system based on deep learning can be improved systematically by training iteratively on a larger number and variety of examples. Furthermore, a pre-trained model (i.e., a trained network) from one domain can be adapted through “fine-tuning”and applied to the same task in a new domain, given that underlying data distributions of both tasks are similar enough. This contrasts to most other learning approaches where the model needs to be completely re-trained when new observations are made available. On the down-side, training a deep neural network from scratch requires massive amounts of annotated data, or data that in some way represent the desired output. Furthermore, the network architecture is often complex, making it difficult to interpret the link between the input data and the predictions. Excellent previous reviews of the broader concepts of deep learning have been presented for medical image analysis Figure 1. Overview of conventional versus deep learning workflows. The human in the center provides input in the form of, for example, parameter tuning and feature engineering in each step of the conventional workflow (black dashed arrows) using annotated data. Conversely, the deep learning workflow requires only annotated data to optimize features automatically. Annotated data is a key component of supervised deep learning as illustrated in the example classification workflow. Other example tasks, as discussed in the text, follow a similar pattern. The example image was provided by the Broad bioimage benchmark collection. Cytometry Part A 95A: 366–380, 2019 367 REVIEW ARTICLE (16,17), health informatics (18), and microscopy (19). The focus of this review is to highlight how deep learning is currently used for image cytometry, including cytology, histopathology, and high-content image-based screening for drug development and discovery. We aim to describe this very quickly emerging branch of image cytometry and explain how it differs from previous “classical”approaches to detect objects, extract features, and/or classify morphological changes, and treatment responses at the microscopic level. We start by defining the key concepts, terms, and vocabulary used in deep learning. We thereafter review the application areas of deep learning in image cytometry, and highlight a series of successful contributions to the field. Then, we discuss challenges and limitations that are often encountered when applying deep learning in image cytometry. Finally, we discuss some of the most recent method developments and highlight novel techniques that have not yet been used for cell analysis in microscopy data, but have the potential to advance the field in the future. As a guide to further reading this review includes a table with a short summary and links to 256 articles on deep learning in cytometry published prior to August 31, 2018, categorized based on imaging modality, task, and biological sample type. OVERVIEW OF KEY CONCEPTS Deep learning architectures come in many flavors, and before going further into the descriptions of deep learning methods, we explain some key concepts: Convolution A convolution can be equivalent to applying a filter kernel to an image, such as a 3×3 mean filter with weight values set to one. To determine the output pixel values, a mean filter slides over the input image. For each pixel position, the kernel weights are multiplied by the pixel values in the corresponding region covered by the kernel, then summed together and divided by the kernel size. In other words, a convolution is a specialized type of linear operation that performs two functions: multiplication and addition. To encode the representations (as illustrated in Fig. 2), an element-wise product between the weights of the kernel and the receptive field of the input image data is calculated at each location and summed to determine the value in the corresponding position of the feature map (or output image). Receptive Field The receptive field (RF) is the extent of the convolution operation where local information (i.e., neighboring pixels) in an image sub-region is taken into account. Locally it is simply the filter size, but in a layered downsampling network it could also mean the region in the input image from which information is propagated. Activation Function An activation function is an operation to threshold the calculated output of a convolution prior to submitting the signal to the next layer of the network. The choice of activation functions has a significant implication on the training process and network performance. Their usage depends on the type of network and also on the type of layer in which they operate. Non-linear operations, such as sigmoid or hyperbolic tangent (tanh) functions, were previously common, but the rectified linear unit (ReLU, (20)) is currently the mainstream activation function since it accelerates the convergence of gradient-based learning and circumvents the problem of vanishing gradients (discussed below). Softmax (21) activation is usually employed in the final layer of a CNN-based classifier, where its output is equivalent to a categorical probability distribution that determines the class of each input image (or pixel). Feature Maps The feature map is the resulting output of the convolution and activation function operations. The dimensions of the resulting feature maps are controlled by three userdefined parameters (often referred to as hyper-parameters): depth, stride, and zero-padding. These hyper-parameters are specified before performing any convolution operations. The depth here corresponds to the number of convolution kernels (depth is sometimes also used for the total number of layers in the network). The stride is the number of pixels by which the kernel shifts over the input image in each step and determines the overlap between individual output pixels. Zero padding is used to circumvent the reduction in output image size produced by convolution operations (by padding the input image with zeros at the borders). Pooling Pooling is an operation to down-sample the output of the convolutions, similar to binning. The most common pooling operation is max pooling, which outputs the maximum value in a local neighborhood of each feature map (as illustrated in Fig. 3), and discards all the other values. It progressively reduces the spatial dimensions of the given feature maps, and thus decreases the number of pixels to process in the next layers of the network, while maintaining information important for the task at hand (8). There are several other pooling operations such as L2-norm pooling and global average pooling (22). Figure 2. An example of an input image Ι 7x7 convolved with a filter k 3x3 with weights of zeros and ones to encode a representation (feature map). The receptive field is highlighted in pink and the corresponding output value for the position is marked in green. [Color figure can be viewed at wileyonlinelibrary.com] 368 Deep Learning in Image Cytometry REVIEW ARTICLE Pooling reduces the complexity of the previously encoded representations, and can thus be seen as a regularization technique that combats the problem of overfitting (see below). Densely (or Fully) Connected Layer In a densely or fully connected layer each neuron of the input layer is connected to every neuron in the succeeding layer, as illustrated in Figure 4a. This combines all representations encoded from the previous layers. For 2D feature maps this is done by flattening the maps into a vector, followed by vector– matrix multiplication. In contrast to the local connection style of convolutional layers, a fully connected layer follows the same connectivity as an MLP. Depending on the complexity of the task, a single or a series of fully connected layers are often added prior to the final classification output layer. Learning and Optimization A process to determine the set of optimal values of trainable parameters (e.g., kernel weights in convolution layers and weights in dense layers), as shown in Figure 4b. These parameters are optimized by minimizing a loss function, such as cross-entropy loss; thus diminishing the discrepancies between the predicted outputs and the given annotations. On a training dataset, the network performance under a particular set of parameters is computed iteratively by a loss function through forward propagation, followed by backpropagation. Backpropagation propagates the loss backward from the output to input layers for computing the gradients of each parameter with respect to the loss. The parameters are then updated using gradient descent optimizers (22). The gradient is a measure of how much the loss changes with respect to changes in parameter values. This iterative process requires many steps, and is the main reason why training a deep neural network requires substantial computational power and time. Although neuroscientists have long rejected the idea that learning through backpropagation is occurring in the human cortex, intriguing possible mechanisms refuting this rejection have been recently suggested (23,24). Overfitting and Underfitting Overfitting occurs when the parameters of a model fittoo closely to the input training data, without capturing the underlying distribution, and thus reducing the model’s ability to generalize to other datasets. Conversely, underfitting is the result of an excessively simple model which is unable to capture the underlying distributions of the input data. Both overfitting and underfitting lead to poor predictions on unseen data. Regularization techniques, as described below, strive to balance overand underfitting, and enable the model to both adequately fit the training data and generalize well to new data. Dropout A regularization technique that reduces the interdependent learning among the neurons to prevent overfitting. Some neurons are randomly “dropped,”or disconnected from other neurons, at every training iteration, removing their influence on the optimization of the other neurons. Dropout creates a sparse network composed of several networks—each trained with a subset of the neurons. This transformation into an ensemble of networks hugely decreases the possibility of overfitting, and can lead to better generalization and increased accuracy (25). Batch Normalization A regularization technique that operates between the layers by continuously taking the output of a particular layer Figure 3. The sub-sampled output of a max-pooling operation with a stride of 2 applied on an input image (I). [Color figure can be viewed at wileyonlinelibrary.com] Figure 4. Learning process of a DNN. (a) A dense layer with an input layer where all the encoded representations from the previous layers are fully connected to the next layers. (b) Zoomed-in view of an example neuron showing the forward propagation to compute the output ȳ, where the non-negative activations are defined using the ReLU. (c) Gradient-decent based optimization of the loss function in a forward/backward propagation. [Color figure can be viewed at wileyonlinelibrary.com] Cytometry Part A 95A: 366–380, 2019 369 REVIEW ARTICLE and normalizing it before sending it across to the next layer. Batch normalization (26) enables the network to learn faster with better generalized performance. When training with batch normalization, each feature map computed by a convolution operation is normalized separately over each batch of samples. Skip (Residual) Connections A skip, or residual connection (27), copies and combines the input of one layer with the output of at least one skipped convolution layer (or block), as illustrated in Figure 5. With an increasing number of layers, the performance of a network may rapidly degrade if the weights become very small during training (referred to as vanishing gradients, (8)). Skip connections allow the gradients to flow freely through possibly hundreds of layers, and thus enable the network to learn evenly across all layers. Data Augmentation An approach to overcome the challenges posed by a limited amount of annotated training data. Augmentation is performed by artificially generating more annotated training data, typically by mirroring and rotating the original images (see section on “Data Considerations in Deep Learning” below). Transfer Learning This is the concept of employing a pre-trained network, which was trained on a large number of samples for a similar task, for a new task with little annotated image data. For instance, one can employ transfer learning between imaging modalities by training a network on phase contrast images and using it on fluorescence images (28) for cell segmentation. DEEP LEARNING METHODS Before delving into how deep learning is used in image cytometry, we briefly describe four general types of deep learning methods. Convolutional neural networks (CNNs) for segmentation and classification of images are often used in a supervised learning setting, meaning that the networks have to be trained using labeled training samples. The networks are built by stacking many convolutional and pooling layers, as shown in Figure 1. The convolution layers encode the discriminative representations/features, and the pooling layers induce a degree of scaleand translationinvariance. The earlier layers typically describe local features that have a small receptive field representing a small and local region of the input image, while deeper layers have larger receptive fields, representing information combined from a larger region in the input image (29). All convolution kernels scan across the entire image, meaning that objects need not be at a specific location to be correctly detected (8,22). Following the final convolution and/or pooling layer the output is flattened and connected to one or more fully connected layers. The neuron(s) in the final output layer give the class probability (either of each input pixel for segmentation or of each input image for classification). Recurrent neural networks (RNNs) build memory into the network by feedback loops, that is, feeding back the output from a neuron as input to itself. The basic RNN consists of “hidden states”that evolve over time through non-linear operations on the previous hidden states and the current inputs. Deep RNNs, that attempt to retain information from the distant past, can be difficult to train due to vanishing gradients. Various solutions to this problem have been proposed, such as skip connections across time. Although RNNs are primarily used for one dimensional sequential data they can also be applied to images, whereby they move across space rather than across time, and can potentially capture more long distance interactions than those caught by CNNs (22). Autoencoders can be used to create a low dimensional representation or “code,”of high dimensional data, much like principal components analysis (PCA), but in a nonlinear manner. An inverse encoder, or “decoder function,”attempts to reconstruct the input from the learned representation (30). Multiple hidden layers can be stacked to the encoder and decoder functions, creating a stacked autoencoder, for learning more complex nonlinear data compression. The learned codes can be made more generalizable using sparsity constraints, which encourage the model to activate fewer neurons (31), or by being trained to “denoise”a noise corrupted version of the input data (32). For image data the fully connected encoder layers are replaced by convolutional and pooling layers, and equivalently the decoder layers by deconvolutional and unpooling layers (33). Generative adversarial networks (GANs), are networks that can generate synthetic/simulated images that closely mimic the original images (34). In its simplest form a GAN can be thought of as instigating a zero-sum game between two networks—a generator and a discriminator. The generator creates counterfeit data samples which it hopes to “fool” the discriminator with, while the discriminator attempts to correctly classify the real and fake samples. Convergence is reached when the discriminator can no longer differentiate between real and fake samples. However, convergence is not always guaranteed and GANs currently require careful architectural and hyper-parameter choices to ensure stability. Generative models can be used to create synthetic datasets, for example, if relatively little annotated data is available (22). Figure 5. An example of a skip connection, connecting the input with the output of one convolution block (consisting of a convolutional layer, a batch normalization layer, and a ReLU activation function). 370 Deep Learning in Image Cytometry REVIEW ARTICLE SURVEY OF PUBLISHED ARTICLES ON DEEP LEARNING FOR IMAGE CYTOMETRY This review is based on a large number of published articles selected based on automated searches in Scopus, Medline, bioRxiv and arXiv, with the search terms “deep learning,” “convolutional neural networks,”“biomedical images,” “microscopy,”and “cells”(in several different combinations) prior to August 31, 2018. We have also included articles from other application areas in cases where we believe the methodological contributions are of significant interest for the field of image cytometry. We excluded articles where handengineered features were used as the network input, and focus on end-to-end representation-based learning methods where pixel data is the only input. The articles can be grouped into general themes based on tasks (as discussed in the section on “Deep Learning for Image Cytometry”) and based on application areas (as discussed in the section on “Applications of Deep Learning Methods”). Many of the articles are referred to in the following text, but we also provide an overview of the articles in Supporting Information Table S1, and as an infographic in Figure 6. An interactive version including short article summaries is accessible at https://anindgupta.github.io/ cyto_review_paper.github.io. The table contains links to each original article, a brief description of key concepts, and a categorization of each article based on tasks (P/S/C): image processing (P), segmentation (S), and classification (C), sample type (T/CL/SB): tissue (T), cells (CL), and subcellular structures (SB), and finally modality (FL/BF/EM): fluorescence (FL), bright field (BF), and electron microscopy (EM). The infographic in Figure 6 serves as a guide to help the reader find articles of interest in relation to a specific task, sample type, or imaging modality, each number in the graphic matching with the corresponding reference. The number of articles in each section can be used to visually assess the current status of the field in terms of popularity and applicability. Given a field of interest, the articles in the corresponding cell of the table become references on the use of deep learning method for this specific task and some of the articles present comparisons of different analysis approaches. It also illustrates the areas of image cytometry where deep learning has been most actively used so far. Note that references to articles not concerned with deep learning for image cytometry are only listed in the regular reference list. DEEP LEARNING FOR IMAGE CYTOMETRY Deep learning can be used for a number of different data processing and analysis tasks, here grouped into image processing, segmentation, classification, and detection. Figure 6. An infographic as a guide to help the readers find articles of interest in relation to a specific task, sample type, or imaging modality, each number in the graphic matching with the corresponding reference. An interactive version linking directly to the source articles can be found at https://anindgupta.github.io/cyto_review_paper.github.io. [Color figure can be viewed at wileyonlinelibrary.com] Cytometry Part A 95A: 366–380, 2019 371 REVIEW ARTICLE Deep Learning for Image Processing Pre-processing of microscopy image data is often necessary to compensate for variability in data collection and to improve downstream analysis. For all of these pre-processing steps both the inputs and outputs are images. Autoencoders can differentiate between true signal and noise, and Su et al. (35) were among the first to exploit them (using an adaptive dictionary and template) to reduce noise and reconstruct cell nuclei in histopathological images. Later, Rivenson et al., (36) presented an adapted U-Net architecture (an autoencoder with skip connections, see section on “Deep Learning for Segmentation”) to computationally improve the resolution of bright field images of tissue samples acquired with a 40×/0.95NA objective to be adjusted such as to achieve a resolution that was equivalent to images acquired with a 100×/1.4NA oil immersion objective. Using a similar approach, Rivenson et al. (37) improved the quality of mobile phone microscopy data to match that of high quality bench-top microscopy (with respect to signal-to-noise ratio, stain normalization, aberrations and distortions). Weigert et al. (38) presented a customized U-Net-based content aware restoration network, and successfully recovered 3D isotropic resolution in fluorescence microscopy data. Such high-quality restoration equates to faster imaging times with less phototoxicity; thus enabling the imaging and analysis of more sensitive samples. Recently, Wang et al. (39) employed GANs to improve the resolution of wide-field fluorescence microscopy images acquired with a 10×/0.4NA objective and achieved a resolution that is equivalent to images acquired with a 20×/0.75NA objective. They also applied their GANs to diffraction-limited confocal images and achieved a 2.6×increase in resolution, corresponding to the resolution of stimulated emission depletion microscopy images. Ouyang et al. (40) combined CNNs and GANs to perform fast super-resolution reconstruction for localization microscopy using a limited number of frames and/or wide-field microscopy images. Nehme et al. (41) employed a fully convolutional encoderdecoder network to achieve fast super-resolution reconstruction in single-molecule localization microscopy, without requiring any prior knowledge of the structure of interest. Color and intensity variations in hematoxylin and eosin (H&E) stained tissue samples from different labs, instruments, and imaging sites may influence CNN performance, as reported by Ciompi et al. (42), and stain normalization prior to training may be necessary. Janowczyk et al. (43) employed autoencoders to achieve unsupervised stain normalization using a novel tissue distribution matching technique for color standardization. Another way to account for stain variability is to include stain differences in the training stage—so called stain or color augmentation. Tellez et al. (44) performed stain augmentation directly on H&E channels for whole slide mitosis detection in Breast Histology, and Arvidsson et al. (45) implemented color augmentation in the HSV space to generalize CNNs for prostate cancer classification for multiple imaging sites. Bentaieb et al. (46) performed stain normalization by style transfer across datasets using a GAN coupled to a CNN for end-to-end histopathology image classification and tissue segmentation. Deep Learning for Segmentation Segmentation, meaning to divide an image to its meaningful parts or objects, requires making accurate local predictions while accounting for global context. One of the first applications of deep learning for segmentation in biomedical image analysis employed a sliding-window CNN as a pixel classifier to segment neuronal membranes in patches of electron microscopy (EM) images (47). This approach involves a trade-off, whereby smaller patches sacrifice contextual information for location accuracy and vice versa. To resolve this, Ronneberger et al. (48) presented a more elegant network architecture, referred to as U-Net, which uses contracting convolving encoder layers (learning global context) skipconnected to expanding up-sampling decoder layers (learning high-resolution location). They also performed random elastic deformations on annotated training data to augment their small training dataset (as described below in the section on “Data Considerations in Deep Learning”). The U-Net architecture achieved excellent results for three different segmentation tasks: neuronal structures in EM images; Glioblastoma– astrocytoma U373 cells in phase contrast microscopy images; and HeLa cells in differential interference contrast (DIC) microscopy images. Cicek et al. (49) extended U-Net to volumetric images with 3D U-Net, which incorporates 3D convolution and pooling operations and is trained using only a small number of annotated 2D slices. Sadanandan et al. (28), showed that the dimensions of U-Net could be reduced by combining raw images with images pre-filtered with task specific hand engineered filters, achieving robust segmentation of Escherichia coli and mouse mammary cells in both phase contrast and fluorescence images. In another article Sadanandan et al. (50) combined CNNs and GANs for segmenting spheroid cell clusters in bright field images. Rather than forcing the GAN to re-create synthetic images, it was recursively used to improve a set of manually drawn segmentation masks, achieving performance gains over a baseline CNN segmentation architecture. Arbelle et al. (51) presented a GAN-based architecture, named Rib Cage, for segmenting H1299 cells in fluorescence microscopy images. They employed a multistream discriminator network taking three channels as inputs: gray level channel of fluorescent images; manually segmented channel of fluorescent images using the Ilastik software (52); and a concatenation channel of the former two channels. Training was performed using a limited amount of annotated examples in a weakly supervised manner. Their unique discriminator network leads to improved performance over other tested architectures. Su et al. (35) achieved improved segmentation performance on both brain tumors and lung cancer cell images using stacked autoencoders. They trained the network using edge enhanced images, centered on cells detected during an initial detection stage, and combined them together with manually annotated cell boundaries. Duggal et al. (53) utilized a form of generative model known as a deep belief network (30) for separating touching or overlapping white blood cell nuclei from leukemia in microscopy images. Recently, Haering et al. (54) presented a cycle-consistent GAN (Cycle-GAN) 372 Deep Learning in Image Cytometry REVIEW ARTICLE for segmenting epithelial cell tissue in drosophila embryos. Their approach circumvents the need for annotated data by employing two generators, where the second generator translates the output of the first generator back to the input space. Despite the fact that the Cycle-GAN is trained without annotated data, it still demonstrated comparable results to U-Net. Deep Learning for Classification and Detection DNNs can be used for classifying images, and also to detect and classify objects within the image. Depending on the available labeled training data, the images or objects can be classified into two or more classes. It is important to note that the confidence in such multi-class predictions may vary depending on the amount of labeled data for each class. Object detection requires both classification and localization of structures. Localization is usually achieved with bounding boxes around the objects of interest, where the outputs are the spatial coordinates of the object, a minimal width and height of the bounding box, and the respective object class. Object detection can be achieved in various ways: (i) regions of interest can be proposed and classified as object or background; (ii) CNN generated feature maps can be used to find bounding boxes; or (iii) networks can be trained endto-end to simultaneously propose bounding boxes and classify objects. The first approach requires a region proposal step prior to classification (55), as used in the Regional CNN (RCNN) model (56). This model, however, is relatively slow as it needs to generate a number of region proposals per image. For faster performance “faster R-CNN”(57) was proposed, comprising only two networks; one for classification and one for region detection. Hung et al. (58) used a faster RCNN architecture to identify and count malaria infected blood cells in bright field images and achieved results comparable to human annotations. Ciresan et al. (59) used a CNN for detecting mitotic cells in H&E stained breast cancer histology images. In this work, the small training dataset was augmented using mirroring and rotations. The CNN outputs a probability map, and as a postprocessing step, a smoothing filter suppressed noise before detecting mitotic events. Similarly, Wang et al. (60) presented a CNN for classifying neutrophil cells on H&E histology tissue images to identify active inflammation. They combined the CNN with a Voronoi diagram of clusters to deal with complex cellular context. Input data was augmented by horizontal mirroring and random cropping. Mao et al. (61) presented a CNN to classify circulating tumor cells in phase contrast microscopy images. In an another work, Durant et al. (62) leveraged CNNs to classify erythrocytes into 10 unique classes. They also performed rotation and mirroring augmentations and achieved a high degree of accuracy for measuring erythrocyte morphology profiles. Recently, Fleury et al. (63) presented a light-weight CNN, MobileNet, to detect and classify blood-borne pathogen images directly on a smartphone. An alternative approach to object detection is end-toend training for proposing and classifying bounding boxes. For example, You Only Look Once (YOLO, (64)), does this by dividing the input image into a rectangular grid and predicting a confidence score for several bounding boxes in each grid and the class probabilities for objects inside each grid. By multiplying the box scores with the class probabilities, one obtains a value per grid-cell that is thresholded to give the final bounding box predictions. This approach is comparably fast since it only needs one pass through the network per image. Another related method is the Single Shot MultiBox detector (65) which utilizes small convolutional filters for bounding box predictions and extracts feature maps at different scales for class predictions. CNNs can also be used for image quality control, discarding poor quality images (e.g., those that are out of focus) prior to further analysis. Work by Yang et al. (66) exploited CNNs on synthetically defocused fluorescence images of DAPI stained nuclei to determine the focus level of image patches. Wei et al. (67) employed a CNN, pre-trained on ImageNet data, to classify the focus of bright field and phase contrast images in numerous z-level classes. They used this approach to automatically control focus during time-lapse microscopy. Transfer Learning and Domain Adaptation Domain adaptation and transfer learning are two related deep learning methods for reusing acquired knowledge. In transfer learning, the features learned by a CNN are reused for different tasks in a similar domain (68) whereas in domain adaptation a discriminative model trained in one domain is applied to the same task in a different domain (7,69). Both approaches are useful when labeled data are lacking, such as in image cytometry, where manual annotations are time consuming to make and require a high level of expertise. For transfer learning, large annotated datasets, like ImageNet (70), can be used to pre-train state-of-the-art CNNs such as Resnet (27) and Inception (71). The transferred parameter values—providing good initial values for gradient descent—can be fine-tuned to fit the target data (72). Alternatively, the pre-trained parameters in the initial layers can be frozen—capturing generic image representations—while the parameters in the final layers can be fine-tuned to the current task (73). Relative to training from scratch, transfer learning allows the fitting of deeper networks, using fewer task-specific annotated images, for improved classification performance and generalizability. The reuse of “off-the-shelf”features in domain adaptation was exemplified by Chacon et al. (74) who combined two U-NET architectures for segmenting mitochondria and synapse EM images. In their case sufficient labeled data was available for one brain region (source domain) but not for another (target domain). However, neural network performance may also degrade under domain adaptation (75). CNNs trained on biomedical images, captured under specific experimental conditions and imaging setups, may have poor generalizability as a consequence of variability in the acquisition and staining processes. This is typical for histology applications. Unsupervised domain adaptation (76) has the potential to resolve this problem. For instance, Yu et al. (77), Cytometry Part A 95A: 366–380, 2019 373 REVIEW ARTICLE 95. Ehteshami Bejnordi B, Mullooly M, Pfeiffer RM, Fan S, Vacek PM, Weaver DL, Herschorn S, Brinton LA, van Ginneken B, Karssemeijer N, et al. Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies. Mod Pathol 2018;31(10):1502–1512. https://doi.org/10. 1038/s41379-018-0073-z. 96. Bychkov D, Linder N, Turkki R, Nordling S, Kovanen PE, Verrill C, Walliander M, Lundin M, Haglund C, Lundin J. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci Rep 2018;8(1):3395. https://doi.org/10.1038/ s41598-018-21758-3. 97. Bandi P, Geessink O, Manson Q, van Dijk M, Balkenhol M, Hermsen M, Bejnordi BE, Lee B, Paeng K, Zhong A, et al. From detection of individual metastases to classification of lymph node status at the patient level: The CAMELYON17 challenge. IEEE Trans Med Imaging 2018;9:1. https://doi.org/10.1109/tmi.2018. 2867350. 98. Qaiser T, Mukherjee A, Reddy PBC, Munugoti SD, Tallam V, Pitkäaho T, et al. HER2 challenge contest: A detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues. Histopathology 2017; 72(2):227–238. https://doi.org/10.1111/his.13333. 99. Sirinukunwattana K, Pluim JPW, Chen H, Qi X, Heng P-A, Guo YB, Wang LY, Matuszewski BJ, Bruni E, Sanchez U, et al. Gland segmentation in colon histology images: The glas challenge contest. Med Image Anal 2017;35:489–502. https://doi. org/10.1016/j.media.2016.08.008. 100. Akram SU, Kannala J, Eklund L, Heikkilä J. Cell Segmentation Proposal Network for Microscopy Image Analysis. Lecture Notes in Computer Science. New York: Springer International Publishing; 2016;21–9. DOI:https://doi.org/10. 1007/978-3-319-46976-8_3 101. Wang M, Ong L-LS, Dauwels J, Asada HH. Multicell migration tracking within angiogenic networks by deep learning-based segmentation and augmented Bayesian filtering. J Med Imaging 2018;5(02):1. https://doi.org/10.1117/1.JMI.5.2.024005. 102. Phan HT, Kumar A, Feng D, Fulham M, Kim J. An unsupervised long short-term memory neural network for event detection in cell videos. arXiv preprint arXiv: 1709.02081; 2017. Available from: https://arxiv.org/abs/1709.02081 103. Villa AG, Salazar A, Stefanini I. Counting cells in time-lapse microscopy using deep neural networks. arXiv preprint arXiv:1801.10443; 2018. Available from: https://arxiv.org/abs/1801.10443 104. Kimmel J, Brack A, Marshall WF. Deep convolutional and recurrent neural networks for cell motility discrimination and prediction. bioRxiv; 2017. DOI: https:// doi.org/10.1101/159202 105. Wang C, Zhang X, Chen Y, Lee K. vU-net: Accurate cell edge segmentation in time-lapse fluorescence live cell images based on convolutional neural network. bioRxiv; 2017. DOI:https://doi.org/10.1101/191858 106. Su Y-T, Lu Y, Chen M, Liu A-A. Spatiotemporal joint mitosis detection using CNN-LSTM network in time-lapse phase contrast microscopy images. IEEE Access 2017;5:18033–18041. https://doi.org/10.1109/ACCESS.2017.2745544. 107. Tellez D, Balkenhol M, Otte-Holler I, van de Loo R, Vogels R, Bult P, Wauters C, Vreuls W, Mol S, Karssemeijer N, et al. Whole-slide mitosis detection in H&E Breast Histology Using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans Med Imaging 2018;37(9):2126–2136. https://doi. org/10.1109/tmi.2018.2820199. 108. Sadanandan SK, Ranefall P, Le Guyader S, Wählby C. Automated training of deep convolutional neural networks for cell segmentation. Sci Rep 2017;7(1):7860. https://doi.org/10.1038/s41598-017-07599-6. 109. Buda M, Maki A, Mazurowski MA. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw 2018;106:249–259. https:// doi.org/10.1016/j.neunet.2018.07.011. 110. Ljosa V, Sokolnicki KL, Carpenter AE. Annotated high-throughput microscopy image sets for validation. Nat Methods 2013;10(5):445. https://doi.org/10.1038/ nmeth0513-445d. 111. Orloff DN, Iwasa JH, Martone ME, Ellisman MH, Kane CM. The cell: An image library-CCDB: A curated repository of microscopy data. Nucleic Acids Res 2012; 41(D1):D1241–D1250. https://doi.org/10.1093/nar/gks1257. 112. Litjens G, Bandi P, Ehteshami Bejnordi B, Geessink O, Balkenhol M, Bult P, Halilovic A, Hermsen M, van de Loo R, Vogels R, et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: The CAMELYON dataset. GigaScience 2018;7(6):1–8. https://doi.org/10.1093/gigascience/giy065. 113. Spjuth O, Bongcam-Rudloff E, Dahlberg J, Dahlö M, Kallio A, Pireddu L, Vezzi F, Korpelainen E. Recommendations on e-infrastructures for next-generation sequencing. GigaScience 2016;5(1):26. https://doi.org/10.1186/s13742-016-0132-7. 114. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M. Tensorflow: A system for large-scale machine learning. OSDI; 2016 (Vol. 16, pp. 265–283). Available from: https://arxiv.org/abs/1605. 08695 115. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T. Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia 2014: 675–678. Available from: https://arxiv.org/abs/1408.5093 116. Team TT, Al-Rfou R, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F, Bayer J, Belikov A, Belopolsky A. Theano: A Python framework for fast computation of mathematical expressions. arXiv preprint arXiv: 1605.02688; 2016. Available from: https://arxiv.org/abs/1605.02688 117. Collobert R, Kavukcuoglu K, Farabet C. Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS workshop 2011. DOI:https://infoscience.epfl. ch/record/192376/files/Collobert_NIPSWORKSHOP_2011.pdf 118. Chen T, Li M, Li Y, Lin M, Wang N, Wang M, Xiao T, Xu B, Zhang C, Zhang Z. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv 2015;1512.01274 Available from: https://arxiv. org/abs/1512.01274. 119. Chollet F Keras: Deep learning library for theano and tensorflow. URL: https:// keras.io/k. 2015;7(8). Available from: https://keras.io/ 120. Kafle K, Kanan C. An analysis of visual question answering algorithms. In: Computer Vision (ICCV), 2017 I.E. International Conference on 2017:1983–1991. Available from: https://arxiv.org/abs/1703.09684 121. Ishaq O, Sadanandan SK, Wählby C Deep Fish. SLAS DISCOVERY: Advancing Life Sciences R&D. Indianapolis, IN: SAGE Publications; 2016:102–107. DOI: https://doi.org/10.1177/1087057116667894 122. Ching T, Himmelstein DS, Beaulieu-Jones BK, Kalinin AA, Do BT, Way GP, Ferrero E, Agapow P-M, Zietz M, Hoffman MM, et al. Opportunities and obstacles for deep learning in biology and medicine. J R Soc Interface 2018;15:20170387. http://rsif.royalsocietypublishing.org/content/15/141/20170387.article-info. 123. Vinyals, O., Toshev, A., Bengio, S., Erhan, D. 2015. Show and tell: A neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3156–3164). DOI: https://arxiv.org/abs/1411.4555 124. Kafle K, Kanan C. An analysis of visual question answering algorithms. In: 2017 I. E. International Conference on Computer Vision (ICCV); 2017; DOI: https://doi. org/10.1109/iccv.2017.217 125. Shin H-C, Le Lu, Kim L, Seff A, Yao J, Summers RM. Interleaved text/image deep mining on a large-scale radiology database. In: 2015 I.E. Conference on Computer Vision and Pattern Recognition (CVPR); 2015; DOI: https://doi.org/10.1109/cvpr. 2015.7298712 126. Mobadersany P, YousefiS, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velazquez Vega JE, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci USA 2017;115:201717139. https://doi. org/10.1101/198010. 127. Kone cný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., Bacon, D.. Federated learning: Strategies for improving communication efficiency. arXiv: 1610.05492; 2016.DOI: https://arxiv.org/abs/1610.05492 128. Turkki R, Linder N, Kovanen P, Pellinen T, Lundin J. Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples. J Pathol 2016;7(1):38. https://doi.org/10. 4103/2153-3539.189703. 129. Armato SG, Roberts RY, Kocherginsky M, Aberle DR, Kazerooni EA, MacMahon H, et al. Assessment of radiologist performance in the detection of lung nodules. Acad Radiol 2009 Jan;16(1):28–38. https://doi.org/10.1016/j.acra. 2008.05.022. 130. Gal, Y., & Ghahramani, Z.. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: International Conference on Machine Learning; 2016:1050–1059. DOI: https://arxiv.org/abs/1506.02142 131. Xie Y, Xing F, Yang L. Deep voting and structured regression for microscopy image analysis. Deep Learn Med Image Anal 2017;155–175. https://doi.org/10. 1016/b978-0-12-810408-8.00009-2. 132. Vovk V. Conditional validity of inductive conformal predictors. Mach Learn 2013; 92(2–3):349–376. https://doi.org/10.1007/s10994-013-5355-6. 133. Su C, Yan Y, Chen S, Wang H. An efficient deep neural networks training framework for robust face recognition. In: 2017 I.E. International Conference on Image Processing (ICIP); 2017 Sep; DOI: https://doi.org/10.1109/icip.2017.8296993 134. Blei DM, Kucukelbir A, McAuliffe JD. Variational inference: A review for statisticians. J Am Stat Assoc 2017;112(518):859–877. DOI. https://doi.org/10. 1080/01621459.2017.1285773. 135. Krueger D, Huang CW, Islam R, Turner R, Lacoste A, Courville A. Bayesian hypernetworks. arXiv:1806.05978; 2017. DOI: https://arxiv.org/abs/1710.04759 136. Xing F, Yang L. Robust Selection-Based Sparse Shape Model for Lung Cancer Image Segmentation. Lecture Notes in Computer Science. New York: Springer Berlin Heidelberg; 2013:404–412. DOI: https://doi.org/10.1007/978-3-642-40760-4_51 137. Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behav Brain Sci 2016;40:e253. https://doi.org/10.1017/ s0140525x1600183. 380 Deep Learning in Image Cytometry REVIEW ARTICLE