Segmentation and 3D Reconstruction of Animal Tissues in Histological Images

Segmentation and 3D Reconstruction of Animal Tissues in Histological Images Master's thesis Liliana Sofia Sales Azevedo Masters in Biomedical Engineering Porto, July 2013 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images Liliana Sofia Sales Azevedo Degree in Biomedical Engineering by Escola Superior de Estudos Industriais e de Gestão (2011) Master's thesis performed under the guidance of:: Supervisor: João Manuel R. S. Tavares Associate Professor of the Mechanical Engineering Department Faculdade de Engenharia da Universidade do Porto Co-Supervisor: Augusto Manuel Rodrigues Faustino Associate Professor of the Pathology and Molecular Immunology Department Instituto de Ciências Biomédicas Abel Salazar Masters in Biomedical Engineering July 2011 IV V "Deus quer, o Homem sonha e a obra nasce." (Fernando Pessoa) Segmentation and 3D Reconstruction of Animal Tissues in Histological Images VI Acknowledgements To Professor and Advisor João Manuel Tavares for the support and monitoring throughout the entire work. To Professor Augusto Faustino for the willingness and dedication. To Alexandre Carvalho for the patience and all the help provided. To Ana Carvalho for the creation of the idea. To Ana Leitão for the important advices and help. To Francisco for the willingness and attention. To Pedro Lhorca for the inspiring thoughts. To my friends André Pinto, Daniel Braga, Daniel Peixoto, Natacha Rosa, Jaime Rodrigues, Sérgio Tavares, and other fellows from office 302, who always encouraged me and helped me in difficult times. To my family, parents and brother for the comprehension and tenderness shown. To all, my heartfelt thanks! Liliana Sofia Sales Azevedo Segmentation and 3D Reconstruction of Animal Tissues in Histological Images VII Abstract Nowadays, the Histology is a science that is still considered the "gold standard" to access anatomical information at a cellular level. Tissue samples are cut into very thin sections, stained, and observed under the microscope by a specialist for pathologic purposes. The study of the tissues structure, cells, acellular components, and their interactions can be useful for the detection and diagnosis of certain pathologies. This fact makes even more interesting to find new techniques and solutions to assist this diagnosis, such as the 3D reconstruction. Based on the extent of biomedical engineering in which this Dissertation can be included, it was considered important to perform a bibliographic review of the histological science, including the types of tissues, the processing techniques of the histological sections, laboratory instruments used as well as meet the basic concepts for image analysis and 3D reconstruction of cell size images. In the present work, a methodology was developed for obtaining three dimensional structures of various tissues. To do so, it was necessary to compare segmentation techniques and combination of image registration methods. Quantitative and qualitative metrics for assessing reconstruction accuracy were also provided. In this Dissertation four experimental cases were analyzed. A preprocessing step was necessary to remove the image noise. The segmentation step proved to be efficient to separate the study object from the image background, and a registration technique accomplished the volume representation. Unlike other projects in tissue reconstruction, this work follows a straightforward methodology with a reasonable computational cost by using the study tool and easily accessibleMatlab2012b®. The knowledge gained in this dissertation creates hopes and goals to achieve in the near future. Keywords: histology, image alignment, image analysis, 3D volume Segmentation and 3D Reconstruction of Animal Tissues in Histological Images VIII Resumo Nos dias de hoje, a Histologia é uma ciência que continua a ser considerada o “gold standard” para aceder a informação anatómica a nível celular. Amostras de tecido são cortadas em secções muito finas, coradas e observadas ao microscópio por um especialista para fins patológicos. O estudo dos tecidos, das células,e dos componentes acelulares e suas interações pode ser útil para a deteção e diagnóstico de determinadas patologias. Tal fato faz com que seja cada vez mais interessante encontrar novas técnicas e soluções que auxiliem neste diagnóstico, como é o caso da reconstrução 3D. Tendo por base o âmbito da Engenharia Biomédica na qual a presente Dissertação pode ser inserida, considerou-se relevante efetuar uma revisão bibliográfica sobre a ciência histológica, incluindo os tipos de tecidos estudados, as técnicas de processamento das secções histológicas, os instrumentos laboratoriais utilizados bem como, reunir os conceitos básicos de análise de imagem e reconstrução 3D de imagens de calibre celular. No presente trabalho, foi desenvolvida uma metodologia para obter estruturas tridimensionais de vários tecidos. Para tal, foi necessário comparar técnicas de segmentação e combinar métodos de alinhamento de imagem. Medidas quantitativas e qualitativas para aceder à precisão da reconstrução foram propostas. Nesta dissertação quatro casos experimentais foram analisados. Uma etapa de preprocessamento foi necessária para remover o ruido presente nas imagens originais. A etapa da segmentação provou ser eficiente em separar o objeto de estudo do fundo da imagem e uma técnica de alinhamento permitiu realizar a representação de um volume. Ao contrário de outros projetos na reconstrução de tecidos, este trabalho segue uma metodologia prática com um custo computacional razoável utilizando a ferramenta de estudo e de fácil acessoMatlab2012b®. O conhecimento adquirido nesta Dissertação, cria esperanças e objetivos a atingir num futuro próximo. Palavras-chave: histologia, alinhamento de imagem, análise de imagem, volume 3D Segmentation and 3D Reconstruction of Animal Tissues in Histological Images IX Contents Acknowledgements ............................................................................................................. VI Abstract .............................................................................................................................. VII Resumo ............................................................................................................................. VIII Contents................................................................................................................................... IX Figure Index ..................................................................................................................... XIII Table Index .................................................................................................................... XVIII Abreviattions and Units Index .......................................................................................... XIX CHAPTER 1 – INTRODUCTION ............................................................................................... 1 1.1 Context ................................................................................................................................ 2 1.2 Goals ................................................................................................................................... 3 1.3 Structure .............................................................................................................................. 4 1.4 Principal Contributions ....................................................................................................... 6 CHAPTER 2 – HISTOLOGY: INTRODUCTION AND STATE OF THE ART ........................ 7 2.1 Introduction ......................................................................................................................... 8 2.2 Tissue Types ........................................................................................................................ 9 2.2.1 Epithelial Tissue ........................................................................................................... 9 2.2.2 Connective Tissue ...................................................................................................... 11 2.2.3 Muscular Tissue ......................................................................................................... 11 2.2.4 Nervous Tissue ........................................................................................................... 12 2.3 Tissue Processing .............................................................................................................. 14 2.3.1 Fixation ...................................................................................................................... 14 2.3.2 Inclusion ..................................................................................................................... 15 2.3.3 Staining ...................................................................................................................... 16 2.4 Microscopic Instruments ................................................................................................... 18 2.4.1 Light Microscopy ....................................................................................................... 18 2.4.2 Electron Microscopy .................................................................................................. 21 2.5 Artifacts and Interpretation Problems ............................................................................... 23 2.6 Summary ........................................................................................................................... 25 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images XVI Figure 5.22 Features with different contrasts does not have match (from Bay et al. 2008). ...... 87 Figure 5.23: Example of an intensity-based Registration between slice 38 and 39: (A) original images and (B) images after the rigid registration (case 1). ........................................................ 89 Figure 5.24: Figure of the left (A) represents the registration is done using only the slice reference; the figure of the right and (B) represents the registration proceeds out from the centre with subsequent images aligned to their neighbors. .................................................................... 90 Figure 5.25: Plot of contour slices in slices number 5, 25, 60 and 100 using a jet colormap (initial case). ................................................................................................................................ 91 Figure 5.26: Reconstruction using a Feature-Based reconstruction with Affine transformation (initial case). ................................................................................................................................ 92 CHAPTER 6 – RESULTS AND DISCUSSION Figure 6.1: (A) Image of slice number 49 and (B) Image of slice number 50 (originals from initial case). ................................................................................................................................. 96 Figure 6.2: (A) Image of slice number 49 and (B) Image of slice number 50, with the same size (initial case). ................................................................................................................................ 97 Figure 6.3: Results of Gaussian filter to the original image (A) with: (B) Gaussian filter with= and (C) Gaussian filter with = (initial case). ................................................ 100 Figure 6.4: Result of the Otsu’s method on (A) original image with rgb2gray and (B) smoothed image with rgb2gray (slice number 60) (initial case). ............................................................... 101 Figure 6.5: Result of the Otsu’s method on (A) saturation of original image and on (B) saturation of smoothed image (slice number 60) (initial case).................................................. 102 Figure 6.6: Binary image from (A) color segmentation of original image and (B) color segmentation of smoothed image (slice number 60) (initial case). ........................................... 103 Figure 6.7: Failed mask using color segmentation by k-means at slice 5 (case 1). ................... 104 Figure 6.8: Failed mask using color segmentation by k-means at slice 26 (case 2). ................. 104 Figure 6.9: Saturation mask for slice 5 (case 1). ....................................................................... 105 Figure 6.10: Saturation mask for slice 26 (case 2). ................................................................... 105 Figure 6.11: Saturation mask for slice 35 (case 3). ................................................................... 105 Figure 6.12: Cleaned image from slice 60 (initial case) (green areas are the missing areas). ... 106 Figure 6.13: Cleaned image from slice 5 (case 1) (green areas are the missing areas). ............ 106 Figure 6.14: Cleaned image from slice 26 (case 2) (green areas are the missing areas). .......... 107 Figure 6.15: Cleaned image from slice 35 (case 3) (green areas are the missing areas). .......... 107 Figure 6.16: Grayscale Image with mask from slice 5 (case 1). ............................................... 108 Figure 6.17: Grayscale Image with mask from slice 35 (case 3). ............................................. 108 Figure 6.18: Sizing: (A) Original image from slice number 70; (B) Sized image from slice 70; (C) Original image from slice number 72 and (D) Sized image from slice 72 (case 2). ........... 109 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images XVII Figure 6.19: Comparison of intensity registration between slices 59 and 60 with different values of iterations. With higher number of iterations like (A) iterations=500) the registration is more perfect than (B) (iterations=1000) - Rigid transformation. (initial case). ................................. 113 Figure 6.20: Intensity-based registration with affine transform (case 2). ................................. 116 Figure 6.21: Intensity-based registration with rigid pairwise transform (case 2). ..................... 117 Figure 6.22: Intensity registration with similarity transform between slice 75 (pink area) and slice 76 (green area). The intersection area is represented by gray intensities (case 3). ........... 118 Figure 6.23: Representation of DICE (blue line) and DICEn (black line) for the Intensity registration with similarity transform (in the red rectangle is noted the abrupt variation of dices values) (case 3). ......................................................................................................................... 118 Figure 6.24: Diagram illustrating the coordinate system around Azimuth (az) and Elevation (el). (adapted from Mathworks 2013u). ............................................................................................ 119 Figure 6.25: Intensity registration based on a similarity transform with viewpoint: View (3) (initial case). .............................................................................................................................. 120 Figure 6.26: Intensity registration based on a similarity transform with viewpoint: View (3) (case 1). ..................................................................................................................................... 120 Figure 6.27: Intensity registration based on a similarity transform with viewpoint: View (3) (case 2). ..................................................................................................................................... 121 Figure 6.28: Intensity registration based on a similarity transform with viewpoint: View (3) (case 3). ..................................................................................................................................... 121 Figure 6.29: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (initial case). ..................................... 123 Figure 6.30: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 1). ............................................ 124 Figure 6.31: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 2). ............................................ 125 Figure 6.32: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 3). ............................................ 126 Figure 6.33: Feature registration based on a similarity transform with viewpoint: View (3) (initial case). .............................................................................................................................. 127 Figure 6.34: Intensity registration based on a similarity transform (REFERENCE) at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 2). .............. 128 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images XVIII Table Index CHAPTER 2 – HISTOLOGY: INTRODUCTION AND STATE OF THE ART Table 2.1: Epithelial Tissue classification according to their characteristics (adapted from HistoVet-UE 2012). .................................................................................................................... 10 Table 2.2: Connective Tissue Classification according to their characteristics (adapted from HistoVet-UE 2012). .................................................................................................................... 11 Table 2.3: Muscular Tissue Classification and its location (adapted from HistoVet-UE 2012). 12 Table 2.4: Nervous Tissue Classification and its location (adapted from HistoVet-UE 2012). .. 12 CHAPTER 4 – IMAGE PREPROCESSING, SEGMENTATION AND REGISTRATION Table 4.1: Classification of registration methodologies (adapted from Maintz and Viergever 1998). .......................................................................................................................................... 48 Table 4.2: Characterization of global models Transformation (adapted from Zitová and Flusser 2003). .......................................................................................................................................... 55 CHAPTER 5 – DATA AND METHODS IMPLEMENTED Table 5.1: Laboratory Tissue Procedures. Materials and time. ................................................... 62 Table 5.2: Total dataset of the cases. .......................................................................................... 68 CHAPTER 6 – RESULTS AND DISCUSSION Table 6.1: Quality parameters for Similarity, Affine and Projective transformations. ............... 98 Table 6.2: Quality parameters for Polynomial of order 2, 3 and 4 transformations. ................... 98 Table 6.3: Final size of the analyzed cases. .............................................................................. 110 Table 6.4: Quality parameters for Similarity and Affine transformations with Feature-Based registration. ................................................................................................................................ 111 Table 6.5: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (initial case). ........................................................................................................... 114 Table 6.6: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 1). .................................................................................................................. 114 Table 6.7: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 2). .................................................................................................................. 114 Table 6.8: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 3). .................................................................................................................. 115 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images XIX Abreviattions and Units Index 2D – Two Dimensions 3D – Three Dimensions CMYK – Cyan, Magenta, Yellow, and Key (black) color space CNS – Central Nervous System CP – Control Points CT – Computed Tomography DNA – Deoxyribonucleic Acid DSC– Dice Similarity Coefficient FLE – Fiducial Localization Error GUI – Graphical User Interface HE – Hematoxylin and Eosin HSI – Hue, Saturation and Intensity color space HSV – Hue, Saturation and Value color space Kg – Kilogram L2RAT – Ratio of the Squared Norm LAB – Lightness and a and b (color-opponent dimensions) space color MAXERR – Maximum Absolute Squared Deviation MB – Megabytes mm – Millimeter MRI – Magnetic Resonance Imaging MSE – Mean Square Error NH 2 – Amidogen nm– Nanometers OPT – Optical Projection Tomography PET – Positron Emission Tomography PH – Potential of Hydrogen PNS – Peripheral Nervous System PSNR – Peak Signal-To-Noise Ratio RANSAC – RANdom SAmple Consensus RGB – Red, Green and Blue components of a color image RMSE – Root Mean Square Error RNA – Ribonucleic Acid ROI – Region of Interest SSD – Sum of Square Differences Segmentation and 3D Reconstruction of Animal Tissues in Histological Images XX SURF – Speeded-Up Robust Features w – Watt µm – Micrometer 1 CHAPTER 1 – INTRODUCTION 1.1 Context 1.2 Goals 1.3 Structure 1.4 Principal Contributions Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 2 1.1 Context Nowadays, Histology is still considered the "gold standard" to access anatomical information at the cellular level. Tissue samples are cut into very thin sections, stained and observed microscopically by an expert, to study normal and diseases processes that involve structural changes like abnormal cellular formations (Rehorek and Smith 2007). Competing alternative techniques to 3D tissue reconstruction are Optical Projection Tomography (OPT), 3D imaging ultrasound, Microscopic Magnetic Resonance Imaging (µm MRI) or Micro Computed Tomography (microCT), among others (Roberts et al. 2012). Although they are mature technologies and are nondestructive imaging techniques, Histology has significant advantage because it allows the conventional staining and interpretation for diagnosing pathologies (Roberts et al. 2012). The Histology involves only the analysis of a few samples or sections of a tissue but there is now an increasing need to observe the tissues on a volumetric point of view, for a fully perception and diagnosis of the tissue. From a geometric volume, it is possible to analyze the micro-architecture of normal tissue (Rehorek and Smith 2007), tumor invasion (Roberts et al. 2012), gene expression (for example, developing mouse embryos (Kaufman et al. 1997) or human brain (Wang et al. 2010). In light of this necessity, it was developed a methodology for obtaining a global 3D model of a tissue. Before any procedure, it was necessary to acquire knowledge about the object to be rebuilt, like the tissue type involved and the histological image processing methods and its consequences. It was also studied image analysis techniques in three main fields that are based on tissue segmentation, registration and subsequent three-dimensional representation of the histological images. Note that before any action, it was necessary to perform an image smoothing in order to remove the image noise resulting from the acquisition process. All these techniques will be explained and justified in the course of this dissertation. The state of the art on similar works served as a guide for developing the reconstruction method. This work, unlike other projects in tissue reconstruction, follows a straightforward methodology with a reasonable computational cost by using the study tool and easily accessibleMatlab2012b®. CHAPTER 1-Introduction 3 1.2 Goals This dissertation aimed to Segment and Rebuild 3D models of various tissues from histological images, developing a methodology of easy implementation in Matlab2012. For this purpose, it was made a bibliographic review on image processing and analysis techniques understood in the work, fundamental tissue types as well the histological image artifacts associated with its acquisition and interpretation. As secondary objectives, it was done a comparing study for the different types of segmentation and alignment methods, more suitable for the images type used. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 4 1.3 Structure In order to make clear and consistent the concerned Dissertation, it was divided into 7 chapters In a summary form it is presented bellow each chapter contents: CHAPTER 2 –HISTOLOGY: INTRODUCTION AND STATE OF ART This chapter presents a description of what is Histology as a studying tissue. Various fundamental tissue types are listed and characterized: Epithelial tissue, Connective tissue, Muscular tissue and Nervous tissue. Here also lies the tissue procedures used to transform the tissues from the original state (fragment) to the final histological image. The materials and devices used for the purpose are described. The electronic or optical microscopes are employed by specialists in order to acquire, analyze and study these images. Although the choice of a suitable microscope is performed according to the tissue type, some artifacts and interpretation problems can arise. CHAPTER 3 –TISSUE 3D RECONSTRUCTION: INTRODUCTION AND STATE OF ART In this chapter, the methodology to create a model of a three-dimensional tissue is generally described and a state of the art of similar living tissue imaging works is provided. These works serve as guides for the development of the work methodology adopted. CHAPTER 4 – IMAGE PREPROCESSING, SEGMENTATION AND REGISTRATION For the development of a methodology with the purpose of analyzing histological images, segment and create a three-dimensional volume from them, it is necessary to have prior knowledge of the existing techniques in the areas of preprocessing, segmentation and registration. This chapter provides such background. CHAPTER 1-Introduction 5 CHAPTER 5 – DATA AND METHODS IMPLEMENTED This chapter discusses more specifically the type of tissues chosen and the specific devices used to obtaining histological image slices. The two approaches followed in the work methodology are described. CHAPTER 6 – DISCUSSION AND RESULTS In this section, the obtained experimental results for the two approaches described in the previous chapter are presented. The optimized parameters are specified for each process, and it is explained the reason for those choices. CHAPTER 7 – CONCLUSIONS AND FUTURE PERSPECTIVES Here it is presented the final conclusions concerning to the experimental results obtained as well as perspectives for future developments. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 12 myofibrils, which contain two types of contractile protein filaments, actin and myosin (Eroschenko 2008). There are three types of muscle tissue in the body: skeletal muscle, cardiac muscle and smooth muscle, and these are responsible for various body functions such as breathing, heart contraction, visceral movements, physical movements of the limbs, among others (Table 2.3) (Eroschenko 2008). Table 2.3: Muscular Tissue Classification and its location (adapted from HistoVet-UE 2012). Muscular Tissue Classification and its location Smooth (involuntary) Digestive Tube and Blood Vessels Skeletal (voluntary) Skeletal Muscle Cardiac (striated and voluntary) Heart (Figure 2.4) 2.2.4 Nervous Tissue The mammalian nervous system is divided into two major parts, the central nervous system (CNS) and the peripheral nervous system (PNS). The CNS consists of the brain and spinal cord. The components of the PNSthe cranial and spinal nervesare located outside the CNS. The nervous system is composed of highly complex intercommunicating networks of nerve cells that receive and conduct impulses for the activation of muscles or glands. The cells of the nervous tissue are the neurons. Generally, each neuron consists of a cell body, numerous dendrites and a single axon. Neuroglia is the matter that supports the neurons and participates in other important functions (Table 2.4) (Eroschenko 2008). Table 2.4: Nervous Tissue Classification and its location (adapted from HistoVet-UE 2012). Nervous Tissue Classification and its location Central Nervous System (CNS) Gray Matter Brain, Spinal Cord (Figure 2.5) White matter Brain, Spinal Cord Peripheral Nervous System (PNS) Nerves Peripheral Nerves Ganglion Retinal cells Nerve endings Limbs Special Receptors Eye, Ear, Nose CHAPTER 2 – Histology: Introduction and State of the Art 13 Figure 2.2: Gallbladder histological cut showing the formation of mucosal folds coated epithelium simple cylindrical (arrow). These epithelial cells have basal nuclei and abundant cytoplasm (adapted from UCPel 2009). Figure 2.4: Striated Cardiac Muscle Tissue with HE technique. The cut shows cardiac muscle fibers irregularly arranged in various orientations and surrounded by connective tissue (adapted from UCPel 2009). Figure 2.3: Dense Connective Tissue (tendon). The slide shows collagen fibers parallel to each other and ordered in a single direction. Between the bundles of fibers are found fibroblasts and fibrocytes. Viewed according to HE technique (adapted from UCPel 2009). Figure 2.5: Cut Spinal Cord. The technique used for coloring this preparation method was Cajal. It is possible to identify the gray matter in the shape of "H" in the center and white matter outside (adapted from UCPel 2009). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 14 2.3 Tissue Processing The preparation of histological sections for microscopic observation is the most commonly used procedure. Because the tissues and organs are too thick to allow the passage of a light beam, they should be cut to provide sections of reduced thickness. Thus, in most cases, tissues and organs are sliced into thin tissue sections and are placed on glass slides. The tissues need to undergo a series of prior treatments for then be sliced by means of high precision instruments, called microtomes (Carneiro and Junqueira 2004). Ideally, one histological sample should preserve in such a way their molecular structure and composition in order to resemble the original body conditions; however, in practice, distortions and losses components are a phenomena that are almost always present, to which is given the name of artifacts (Carneiro and Junqueira 2004). Figure 2.6 shows the procedures applied to tissues from the original state (fragment) to the final histological observation by microscopy. 2.3.1 Fixation In order to preserve the structure and prevent the tissues digestion by the enzymes present inside cells (autolysis) or by bacteria, the tissue fragments should be treated as soon as possible after their removal. This treatment is called Fixation. This process can be carried out using chemical or physical methods (less common) (Bioaula 2007; Carneiro and Junqueira 2004). In the chemical fixation, the tissues are immersed in solutions with denaturing agents or agents that facilitate the stabilization of the molecules forming bridges with neighboring molecules. These solutions are called fixators. As the diffusion of the fixator to the interior of tissues is a time-consuming process, the tissues fragments are typically cut into smaller pieces before being immersed in the fixative in order to facilitate the agent's penetration in tissues and thus enable a good conservation of the structure. Another alternative consists in making an intravascular infusion of the fixator which, through blood vessels, reaches the innermost zones of tissues, resulting in better fixation (Bioaula 2007; Carneiro and Junqueira 2004). One of the best fixators commonly used is a solution of formaldehyde, called formalin. Formaldehyde is a compound which reacts with amino groups (NH 2 ) of tissue proteins (Carneiro and Junqueira 2004). CHAPTER 2 – Histology: Introduction and State of the Art 15 After the fragments fixation with formaldehyde, the samples pass through an ascending series of alcohol solutions composed of (a range of 75 to 100%), from the less concentrated to the most concentrated solution, in order to dehydrate the tissue (Carneiro and Junqueira 2004). 2.3.2 Inclusion To obtain thinner sections of the structures with the microtome, it is necessary that these will be placed in a substance which gives them a certain rigid consistency of all the substances with these properties, paraffin wax and certain plastic resins are the most used (Bioaula, 2007; Carneiro and Junqueira, 2004). The technical process to impregnate the tissue with paraffin is called inclusion or soaking, and it is usually preceded by two steps: dehydration and clearing. In the first phase, water is extracted as the fragments pass through various baths composed of solutions with increasing ethanol concentrations in water (usually ethanol from 70 to 100%). After dehydration, the ethanol present in the fragments is substituted by an intermediate solution (usually an organic solvent) which is miscible both in ethanol and in the enclosing mean (paraffin or resin) (Bioaula 2007; Carneiro and Junqueira 2004). Figure 2.6: Diagram of laboratory tissue preparation procedures. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 16 Xylol is the most widely used intermediate substance. This solvent aims to remove the alcohol from the tissues interior, since the alcohol is not miscible with the paraffin. After this procedure called clearing, the tissues become transparent and / or translucent (Bioaula, 2007; Carneiro and Junqueira, 2004). The next step consists in placing the pieces into molds to be filled with paraffin previously melted, usually 58-60 ° C. The heat causes the evaporation of the solvent, and the empty spaces within the tissue are filled with paraffin. At the end, the tissues embedded in paraffin become rigid after being removed from the stove (Bioaula 2007; Carneiro and Junqueira 2004). The rigid blocks containing the tissues are then taken to a microtome (Figure 2.7), where they are cut by a steel or glass blade, in order to provide cuts 1-10 mm thick. After being sectioned, the slices are placed to float upon a heated water surface. Here the slices are collected by thin glass plates where they will adhere (Carneiro and Junqueira 2004). A completely different way of preparing tissue sections can be accomplished by submitting the tissues to a fast freezing. The fast freezing leaves the rigid tissue, ready to be cut without requiring a phase of dehydration and inclusion. The micrometer used is called a cryostat and was developed for frozen tissue production (Carneiro and Junqueira 2004). This method is very useful for histochemical study of enzymes highly sensitive since the freezing, unlike the chemical fixation, does not inactivate most of the enzymes and the small molecules are maintained in their original locations. It is also a suitable method for the lipids study present in the tissues, since tissue immersion in solvents such as xylene(xilol) dissolves lipidic substances (Carneiro and Junqueira 2004). 2.3.3 Staining Histological cuts should be stained for being analyzed under a microscope, as most of their tissues become colorless as a result of previous processes (Carneiro and Junqueira 2004). Staining methods were developed not only to make the different components of the tissues obvious but also facilitate the distinction between them. The coloring agents behave as acidic or basic compounds and tend to form electrostatic bonds (saline) with tissues ionized radicals (Carneiro and Junqueira 2004). Toluidine (Figure 2.8) and methylene blue are two basic dyes examples. Toluidine blue behaves as a basic dye, binding to basophilic structures of tissues. The main components which ionize and react with basic dyes are those that have acids in their composition acids, for example, nucleic acids, glycosaminoglycans and acidic glycoproteins (Carneiro and Junqueira 2004). CHAPTER 2 – Histology: Introduction and State of the Art 17 Acid dyes such as Orange G, Eosin and Acid Fuchsin mainly stained the components: acidophilus tissue as mitochondria, secretory granules, cytoplasmic proteins and collagen (Carneiro and Junqueira 2004). Of all the colors, the combination of hematoxylin and eosin (HE) is the most used. The hematoxylin blue (or violet) stains the cell nucleus and other acid structures such as cytoplasm portions rich in RNA and hyaline cartilage matrix (Figure 2.9). On the other hand, eosin stains the cytoplasm and collagen in color pink. Mallory (Figure 2.10) and Masson dyes, which are of the trichrome type, which enables the nucleus and cytoplasm observation and helps differentiate collagen of smooth muscle (Carneiro and Junqueira 2004). Although most colorants are useful for tissue components visualization, they provide little information about the chemical nature of the tissue components. So in many procedures, resorts to Immunohistochemistry for more information about a given tissue fragment (Carneiro and Junqueira 2004). The entire procedure, from fixation to observation of a tissue can range from 12h to 2.5 days. The fragment size, type of fixative, and the enclosing mean are parameters influencing the time of the whole process (Carneiro and Junqueira 2004). Figure 2.7: Example of a microtome, a slicing instrument used to slice the tissue in the form of fine cuts, for facilitating their microscope analysis (from Harlow Scientific 2013). Figure 2.8: Toluidine Blue staining with PH = 2.5 from a histological cut of a knee junction (adapted from Histology-World 2012a). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 18 2.4 Microscopic Instruments After the tissue processing, in order to study and analyze the stained sections, Electronic or Optical Microscopes combined with imaging technologies are employed. The choice of a suitable microscope is performed according to the tissue or organelles type that will be studied. The Microscopy equipment can be divided into two major groups: the Light Microscopy (light beam) and Electronic Microscopy (electron beam). 2.4.1 Light Microscopy The Optical Microscope or Light Microscope (Figure 2.11) is the most commonly used to expand the tissue structures and produce a histological picture of high resolution (He et al. 2009). It is composed of by a mechanical component and an optical component (Carneiro and Junqueira 2004). Figure 2.9: Histological slice of the large intestine (adapted from Histology-World 2012b). Figure 2.10: Mallory staining for a histological cut of skin (adapted from Histology-World 2012c). CHAPTER 2 – Histology: Introduction and State of the Art 19 The optical components are composed by three elements: the condenser, the objective(s) and the ocular. The condenser focuses the light and projects a beam. The objective lens produces an enlarged image of the sample towards the ocular, which in return expands again the image and projects it on the retina (Carneiro and Junqueira 2004). The obtained images have a magnification equal to the product of the magnification of the objective by the magnification of the ocular (Carneiro and Junqueira 2004). There are four major groups of light microscopes: Confocal Microscopes, Fluorescence Microscopes, Hiperespetral and Multiespetral Microscopes (He et al. 2009). The Fluorescence and Confocal Microscopes composition and functioning are explained in the following. Fluorescence Microscope When a substrate is irradiated by light with a certain wavelength, it will emit light with a longer wavelength. This phenomenon is called fluorescence (Carneiro and Junqueira 2004). In Fluorescence Microscopy, sections are irradiated with intense ultraviolet light and, in turn, emit light in the visible portion of the spectrum, which makes the fluorescent sample appear bright on a dark background. The fluorescence microscopes either control the intensity of the emitted light, either the light irrradiated by the tissue (Carneiro and Junqueira 2004). Like most tissues do not fluoresce by itself, a fluorescent molecule is required to mark the objects of interest, which can be molecules or subcellular components (He et al. 2009). For example, the dye acridine orange can be combined with DNA or RNA. When observed by the fluorescence microscope, the complex DNAacridine orange fluoresces yellow-green color and the complex RNAacridine orange fluoresces red-orange. This way, it is possible to identify and locate two kinds of nucleic acids into cells (Figure 2.12); a process impossible to achieve using only naked eye observation. Confocal Microscope Because the focus depth of an Optical Microscope is relatively long, the observed image corresponds to the total planes of focus obtainable from a sample with a certain thickness. Considering the tissue section as a three-dimensional object, the Confocal Microscope makes possible to focus only on certain plan. The basic principle of this type of microscope starts with Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 20 a laser beam focused on the sample sectioned. A small hole allows selecting only the image plane of focus. This image reaches the detector and is amplified for observation (Figure 2.13) (Carneiro and Junqueira 2004; He et al. 2009). Figure 2.11: Optical Microscope configuration (from Free Info Society 2013). Figure 2.12: Photomicrograph of kidney cells culture from a hamster stained with acridine orange. The DNA (the nuclei) emits yellow light and the RNA (in the cytoplasm) has reddish color (adapted from Carneiro and Junqueira 2004). CHAPTER 2 – Histology: Introduction and State of the Art 21 2.4.2 Electron Microscopy Due to aberrations and optic diffraction, the image offered by a Light Microscope is not always exactly identical to the original. In addition, the wavelength of visible light limits the resolution (up to 200nm), which allows differentiate cells but difficult the observation of small cellular details (He et al. 2009). This technology called electronic microscopy applies an electron beam to "light up" the tissue, whose wavelength is smaller (about 1nm). The use of this type of microscope requires that the tissues are fixed in a rigid medium such as paraffin, and examined in a vacuum chamber (He et al. 2009). There are two types of electron microscopes: the Transmission Electron Microscope and the Scanning Electron Microscope. Figure 2.13: Illustrative Scheme of the Confocal Microscope functioning (adapted from Carneiro and Junqueira 2004). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 28 3.1 Introduction The Bioimaging is playing more and more a central role in addressing challenges that arise in biology. The post-genomic era arrived and with the human genome sequenced, the focus has to do now with understand the genes role and discover molecular structures that regulate them (Cooper 2009). A realistic picture of phenomena such as cancer requires more than the observation of molecular behavior at the surrounding tissues. Information obtained at a cellular scale is useful for understanding biological mechanisms, intracellular regulation as well as intercellular interaction (Cooper 2009). Thus, imaging such as 3D computational reconstruction, becomes a valuable tool to understand these phenomena. However, the implementation of imaging techniques in Histology has not always been easy. The histological images in certain cases are considered 'noise' for being formed by very indistinguishable repeated structures, like cells. The overall textured look challenges the images alignment and segmentation. The large size of microscopic images also presents a challenge in terms of computation (Cooper 2009). Furthermore, the artifacts used in the image processing are factors to consider. The increasingly number of challenges proposed by the recent biology and technology makes the present day favorable for the development of new systems for image reconstruction and analysis. This chapter describes a general methodology to reconstruct a tissue and presents a state of art of various previous works made until this moment in this field. Four approaches are mentioned: a 3D Reconstruction using 2D histological images Registration, a 3D Reconstruction using a 3D reference, a 3D reconstruction using Cross-slice Regularization and a Smooth-guided 3D Reconstruction based approach. 3.2 State of Art The reconstruction of tissues three-dimensional (3D) structure from a series of sections was always at least in theory, a valuable tool to expand the 'hidden' microscopy dimension in order to study the tissues, cells and subcellular components structure (Xu et al. 2004). Originally, the histological sections were visually examined and were created hand-drawn diagrams. Relying in the ability of the researcher or artist trained biologist. Subsequently, latest CHAPTER 3 – 3D Tissue Reconstruction: Introduction and State of Art 29 techniques in the micrographies and scanned images preparation, introduced a more technical and precise science of 3D reconstruction (Rehorek and Smith 2007). In 1988 is reported the first work in the area of 3D reconstruction from histological techniques. The objective was to reconstruct 3D images of the brain (Hibbard and Hawkins 1988). The histological image reconstruction is now used for a better understanding of the structure and growth pattern, usually associated with diseases (Bussolati et al. 2005). 3.2.1 General Methodologies One of the problems of microscopic image analyses, more specifically with histological images, is that the providing information is only on a (2D) plane, making difficult the understanding of the phenomena and structural relationship between different tissue constituents with the exterior. In mass, the histological manual image analysis is tiring for a single individual and takes a long time. Another disadvantage is that the interpretation of the same histological picture can vary from individual to individual (Cooper 2009). The main approach to obtain 3D information is the tissue reconstruction from a series of consecutive images through "image registration (image alignment)". A particular fragment or specimen undergoes a series of processes (already described in Chapter 2) to obtain the slides with the histological images. The following images are scanned and then are aligned to generate a volumetric dataset of the original tissue (Figure 3.1) (Cooper 2009). Figure 3.1: Illustrative scheme of processes going from the tissue to the 3D reconstruction and subsequent reconstruction and analysis (from Cooper 2009). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 30 It is necessary having an absolute knowledge about the tissue procedures and deformation processes that the type of original tissue in particular undergoes, avoiding reconstruction errors caused by a bad choice of the transformation models in the alignment. A key factor in tissue reconstruction is to identify its boundaries. This process may help define the morphology of specific tissue layers, calculate areas, but also distinguish different regions when a tissue appears to be homogeneous and without salient features. This factor can be solved with an approach called Segmentation (Cooper 2009). This method can rely on clues such as: color, texture and shape (Figure 3.2). Its difficulty increases with the specificity given to the algorithm (Muthu Rama Krishnan et al. 2011). 3.2.2 3D Reconstruction using 2D histological images Registration The registration of consecutive pairs of slices (2D images) from a histological image stack allows creating a coherent 3D geometrical registration for reconstruction. A common approach is to align consecutive pairs of slices according to a slice reference, usually chosen from an original dataset (Malandain et al. 2004; Ourselin et al. 2001a). Multiple registration techniques applied to 2D histological images have been proposed in the literature. The manual techniques or manual registration relied on researcher’s anatomical knowledge to align the images manually. These techniques had the disadvantage of being too slow, operator dependent and with low accuracy (Deverell et al. 1993; Rydmark et al. 1992). The fiducial techniques, such as the name indicates, use fiducial markers to enhance the registration performance (Ford-Holevinski et al. 1991; Goldszal et al. 1996; Goldszal et al. 1995; Humm et al. 1995). The most common procedure uses extrinsic markers (like a rigid Figure 3.2: The visual clues that distinguish tissues include: color, shape, texture and scale. The green line indicates the boundary between two layers of tissue with similar visual appearance (from Cooper 2009). CHAPTER 3 – 3D Tissue Reconstruction: Introduction and State of Art 31 needle). These needles are placed before the sectioning process to create holes and registration is done by overlapping these marks in order to create a volume. However, this technique is an invasive procedure that may damage the specimen; if needles are not correctly perpendicular placed to the cutting or the holes collapse, the 3D reconstruction is compromised (Simonetti et al. 2006; Streicher et al. 1997). In contrast to fiducial techniques, geometric methods based or features methods based are noninvasive and consist of extracting features as points (Guest et al. 2001; Rangarajan et al. 1997), vertices (Kim et al. 1995) or contours (Cohen et al. 1998; Zhao et al. 1993), to the respective slices to perform the registration. Here the registration efficiency dependent on the quality of the feature extraction step. The intensity based methods become quite popular for their accuracy. For example, Kim et al. (1997) proposes a technique using mutual information and Ourselin et al. (2001b) algorithm uses a robust "block-matching" method. There are also methods based on align images principal axes; however, these have become obsolete because they were unable to handle artifacts resulting from histological processing (Hibbard and Hawkins 1988; Schormann and Zilles 1997). 3.2.3 3D Reconstruction using a 3D reference Despite the various methods to perform pair wise slice registration, it was noted that the obtained volume was far from resemble the real anatomical volume. The 3D organ or structure conformation is lost after the tissue sectioning process, and the typical registration algorithm accumulates errors. Visually, the reconstructed volume exhibited a wave pattern along the cutting axis, referred to as "the banana problem" in Malandain et al. (2004) or the "z-shift effect" described by Yushkevich et al. (2006). To solve this problem, some methods use external information of the 3D structures. This morphological information may be acquired using imaging modalities such as MRI, PET, or CT (Ourselin et al. 2001a; Pitiot et al. 2007; Toga et al.1997) or specimen photographic images taken during the sectioning process, called "block-face” (Bardinet et al. 2002; Kim et al. 1997). These methods aim to reconstruct a coherent 3D tissue model through a pair wise registration. Thereafter, the built volume is co-aligned with the chosen modality (e.g. MRI). Dauguet et al. (2007) created a geometric volume from the images 'block-face'. The reconstruction was obtained by alignment the histological images, with reference to the volume obtained previously. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 32 3.2.4 3D reconstruction using Cross-slice Regularization In certain cases, an external reference anatomical is not available. Several methods are proposed in the literature to alleviate reconstruction pair wise limitations. The extension of the pair wise alignment for a larger neighborhood (Yushkevich et al. 2006) or the whole images (Guest and Baldock 1995; Tan et al. 2007; Wirtz et al. 2004) are two options available. For example, Yushkevich et al. (2006) propose a theoretical technique where each slice is rigidly aligned with several slices of the neighborhood, up to 5 slices away, instead of just being aligned with the previous or posterior slice. A weighted graph is constructed where the vertices represent the slices, the edges represent registrations between the slices, and the edge weights represent the misregistration error. A rigid transformation is performed based on the shortest path from every vertex in the graph to the slice reference. Other methods handle the entire set of slices, as is the case of Wirtz et al. (2004). In this study the registration is modelled as a minimization problem of a functional consisting of a distance measure (sum of squared differences), and a regularizer based on the elastic potential of the displacement field. The minimization of the functional leads to the so called Navier-Lamé equations. In Tan et al. (2007) the 3D registration is obtained through the histological similarity matching between neighboring slices. Further optimization is achieved by applying affine transformation to each slice according to feature curve smoothing. At last, Guest et al. (1995) describes an automatic method for correcting the sectioning distortions (warping) in order to produce a smooth object. Each section is modelled as a thin elastic plate where the correspondences between sections are calculated using edge matching points. The finite element method is used to establish the equilibrium position of the elastic transforms. 3.2.5 Smooth-guided 3D Reconstruction In certain cases, the quality of the 3D reconstruction is usually evaluated taking into account the smoothness of the reconstructed structures of interest in the cutting direction of the axis (Guest and Baldock 1995; Ju et al. 2006; Laissue et al. 2008; Tan et al. 2007; Wirtz et al. 2004). In Wirtz et al. (2004) an expert visually examined the smoothness of the edges which delimits the anatomical structures of a mouse brain histological volume and quantified the recognizable structures such as cerebellar fissures, molecular and granular layer and cerebellum white substance. CHAPTER 3 – 3D Tissue Reconstruction: Introduction and State of Art 33 Guest et al. (1995) quantified the smoothing degree of the reconstruction based on the closeness of the corresponding points in each slice must be located close to each other. Ju et al. (2006) proposes a similar procedure to the previous one where the smoothing is evaluated according to the location of the corresponding points in the 2D pair wise. Recently, Laissue et al. (2008) evaluated the alignment errors using the distance measured between the lateral ventricles (crest-lines) identified in the histological volume and in the MRI volume. 5.3 Summary Bioimaging plays a central role in addressing challenges of the recent biology, and the technology makes the present day favorable for the development of new systems for image reconstruction and analysis. The reconstruction of tissues three-dimensional (3D) structure from a series of 2D sections is at least in theory, a valuable tool to expand the 'hidden' microscopy dimension in order to study the tissues and cells. The main approach to obtain 3D information is the tissue reconstruction from a series of consecutive images. To do so, a particular fragment or specimen undergoes a series of processes (described in chapter 2) to obtain the slides with the histological images. The following images are scanned and then are aligned (registration) to generate a volumetric dataset of the original tissue. Four approaches were explained in detail with work examples: a 3D Reconstruction using 2D histological image (registration of consecutive pairs of slices using fiducial markers, noninvasive feature methods or intensity methods), a 3D Reconstruction using a 3D reference (uses external information of the 3D structures to resemble the histological volume to the real one), a 3D reconstruction using Cross-slice Regularization (extension of the pair wise alignment for a larger neighborhood for improving the volume) and a Smooth-guided 3D Reconstruction (reconstruction guided by the smoothness of the reconstructed structures in the cutting direction of the axis). The careful study of these previous projects served as a guide for the methodology developed during this Dissertation. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 34 35 CHAPTER 4 – IMAGE PREPROCESSING, SEGMENTATION AND REGISTRATION 4.1 Introduction 4.2 Preprocessing 4.2.1 Spatial Filtering 4.2.2 Intensity Transformations 4.2.3 Color Spaces 4.3 Segmentation 4.3.1 Global Thresholding 4.3.2 K-means Clustering 4.4 Registration 4.4.1 Registration Methodologies 4.4.2 Transformation models 4.4.3 Registration Accuracy Assessment 4.5 Summary Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 36 4.1 Introduction A common imaging preprocessing step relies on the image enhancement, which is the process of adjusting digital images so that the results are more suitable for further analysis. Removing noise, image smoothening, intensity transformations and manipulation of color spaces makes easier the identification of key features. The Segmentation is a process that subdivides an image into its constituent regions or objects. The procedure stops when the object(s) of interest are isolated. Segmentation is one of the most difficult tasks in image processing since it determines the success or failure of the analysis procedures (Gonzalez et al. 2004). The image registration is the process of overlapping two or more images of the same scene taken at different time points, distinct views and/or by different sensors. It aligns geometrically two images: the reference image (fixed image) and the sensed image (moving image) (Zitová and Flusser 2003). For the development of a methodology for analyzing histological images, segmenting and building a three-dimensional volume of them, it is necessary to have prior knowledge of the existing techniques in the areas of preprocessing, segmentation and registration. Thus, it is made a review of existing preprocessing methods, including spatial filtering and intensity transformations. It is also made a summary of the existing color spaces, since the histological images belong to that dimension. Two main methods of segmentation are briefly listed: the global thresholding and k-means clustering. Finally, it is presented an introduction about image registration that includes methods of alignment, principal alignment approaches (feature-based and intensity-based), transformation models to apply and some techniques to assess registration accuracy. CHAPTER 4 – Image Preprocessing, Segmentation and Registration 37 4.2 Preprocessing In case of poor quality of the input images, for example, due to severe noise, low intensity contrast with weak edges, and intensity inhomogeneity, some preprocessing techniques, like image smoothing, denoising and intensity transformations, may be applied for image restoration (He et al. 2009). Image smoothing is directly related to Spatial Filtering to highlight the major image structure by removing image noise and fine details (He et al. 2009). Intensity Transformations are applied to gray-scale images when becomes important to enhance the contrast of the image (Gonzalez et al. 2004) and color information when available, can be a significant tool to identify the object of interest (INSTRUMENTS 2006). 4.2.1 Spatial Filtering Digital images are prone to a variety of types of noise. Noise is the result of errors in the image acquisition process that result in pixel values that do not reflect the true intensities of the real scene (Mathworks 2013a). Spatial domain techniques operate directly on the pixels of an image. They are usually used to noise reduction. Its methodology follows four steps: (1) defining a center point (,), (2) performing an operation that involves only the pixels in a predefined neighborhood about that center point, (3) letting the result of that operation be the response of that point and (4) repeating the process for every point in the image. If the transformation on the pixels of the neighborhoods is linear, the operation is called linear special filtering (or spatial convolution); otherwise, it is called nonlinear spatial filtering (Gonzalez et al. 2004). Three methods of special filtering are described: Mean, Median and Gaussian filtering. Mean filtering A mean filtering (linear) is a simple, intuitive and easy to implement method of smoothing images, i.e., reducing the amount of intensity variation between one pixel and the next. It is often used to reduce noise in images. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 44 (A) (B) (C) (D) Figure 4.9: Representation of HSV components of an RGB image (A): (B) Hue; (C) Saturation and (D) Value components (adapted from Mathworks 2013d). L*a*b space L*a*b* color space is based on CIE XYZ color space coordinates. This color space consists of a luminosity layer 'L*', chromaticity-layer 'a*'(indicating where color falls along the red-green axis) and chromaticity-layer 'b*' (indicating where the color falls along the blue-yellow axis). Figure 4.10 shows the components of L*a*b space of an example image. CHAPTER 4 – Image Preprocessing, Segmentation and Registration 45 (A) (B) (C) (D) Figure 4.10: Representation of L*a*b components of an RGB image (A): (B) Luminosity layer; (C) red-green axis and (D) yellow-blue axis (adapted from Mathworks 2013d). 4.3 Segmentation Image segmentation extracts objects/regions of interest from the background; these objects and regions are the focus for further disease identification and classification (He et al. 2009). The segmentation algorithms can be classified into three main types: based on threshold, the ones based on clustering techniques and the ones based on deformable models (Zhen et al. 2010). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 46 A theoretical review will be made of segmentation techniques based on global threshold and kmeans clustering in the following. 4.3.1 Global Thresholding Image Thresholding enjoys a central position in applications of image segmentation because of its intuitive properties and simplicity of implementation. Supposing that the histogram in a given figure corresponds to an image %(,) composed by bright objects on dark background, in such way that object and background pixels intensity can be grouped into fields. One simple way to separate/extract the objects from the background is select a threshold T that separates the two classes (Figure 4.11). Any point (,) above or equal to value  is called object point and is called background point when its value is below  (Equation 4.4). When  is a constant, the threshold is named global threshold (Gonzalez et al. 2004): . (  ,  ) = / 1  +%  % (  ,  ) ≥   ( &123)  &+4) ) 0  +%  % (  ,  ) <  ( 1637.&84! ) , (4.4) The Otsu method is a histogram-based method were the gray-level histogram is treated as a probability distribution: 9  :  ; < = 4 ; 4 ,     = = 0 , 1 , 2 , > − 1 (4.5) where 4 is the total number of image pixels, 4 ; the number of pixels that have the intensity level  ; and > the number of possible intensity levels (Gonzalez et al. 2004). Supposing a Figure 4.11: Selecting a threshold by visually analyzing a bimodal histogram (adapted from Gonzalez RC. et al. 2004). CHAPTER 4 – Image Preprocessing, Segmentation and Registration 47 dichotomization of pixels into two classes @ " and @ A , the Otsu’s method chooses the threshold value k that maximizes the variance between-class,  B  , which is defined as:  B  = ꙍ " ( D " − D E )  + ꙍ A ( D A − D E )  (4.6) where ꙍ " and ꙍ A are the probability of the classes occurrence and D " and D A are the classes mean levels of @ " and @ A , respectively (Gonzalez et al. 2004; Otsu 1979). 4.3.2 K-means Clustering This approach is one of the simplest and widely used when it is necessary to extract objects of different classes. Thus, it can be considered a multilevel thresholding (He et al. 2009). Clustering is a way to separate groups of objects and K-means clustering treats each object as having a location in space. It finds partitions such that objects within each cluster are as close to each other as possible, and as far from objects in other clusters. K-means groups image points into clusters by minimizing the objective function (He et al. 2009). This function was the form: G G | I J − D K |  J ∈ M N O K P A (4.7) where I J is the intensity of the image point x i in the class Q K and D K is the mean of points in the class Q K (He et al. 2009). 4.4 Registration Image registration is the process of overlapping two or more images of the same scene taken at different time points, different views and/or by different sensors. Geometrically aligns two images - the reference image (fixed image) and the sensed image (moving image) (Zitová and Flusser 2003). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 48 Registration is a crucial step in all image analysis tasks like image fusion, difference detection and multichannel image restoration. The goal of image registration is to find the optimal transformation that better aligns the structures of interest in the input images (Zitová and Flusser 2003; Oliveira and Tavares 2012). Is typically used in systems analysis and image processing as remote sensing (e.g. multispectral classification, environmental monitoring, detection of discrepancies, images union, weather forecast, creation of high resolution images), cartography (in update maps), computer vision (for target location and automatic quality control), medicine (a combination of PET and CT/MRI images is used to obtain more detailed information about the patient, monitoring tumor growth or compare patient data with an atlas anatomical atlas) and Histology (aiming to rebuild tissue or other structures in 3D to investigate cellular structures) (Zitová and Flusser 2003; Oliveira and Tavares 2012). 4.4.1 Registration Methodologies There are several methods for registering images which can be classified in different ways. Maintz and Viergever (1998) suggested a diagram that provides a good dimensional categorization. These criteria are: Dimensionality, Registration Basis, Nature of the transformation, the Transformation domain, Degree of interaction, Optimization Procedures, Modalities, Subject and Object (Table 4.1). Table 4.1: Classification of registration methodologies (adapted from Maintz and Viergever 1998). Criteria Characteristics Dimensionality Refers to the geometric dimensions of the images to register (2D/2D, 2D/3D, 3D/3D). Registration Basis Corresponds to the points characteristics used to register two/more images. These points can be: Extrinsic: external to the object to align, e.g. fiducial markers, "Stereotactic frame", among others (Cifor et al, 2011); Intrinsic: anatomical features of the object, such as lines and curves (Cifor et al, 2011); CHAPTER 4 – Image Preprocessing, Segmentation and Registration 49 Nature of the transformation Is related to the system of coordinates that define the space. The nature divided into: Rigid; Affine; Projective; Curved. Domain of the transformation Related to the amount of information used in the image: Local: only part of the voxels of the region of interest (ROI) are used; Global: are used all voxels in the region of interest. Iteraction Degree In reference to the registration algorithm control by the operator. Divided into: Automatic; Parameter inticialization; Parameter adjustment along time. Optimisation Procedure Algorithm Approach: the registration quality is estimated continuously during the process, in terms of mapping functions between images and between them. Therefore, may be: Automatic; Iterative search. Modalities Related with fusion of information. Divided in: Monomodal: registration with images of the same modality (for example, MRI-MRI); Multimodal: registration with images of different modalities (for example, MRI-PET or MRI-histological atlas) ( Li et al. 2009; Osechinskiy and Kruggel 2011). Subject Linked with the pacient envolvement: Intrasubject: same subject; Intersubject: Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 50 different subjects; Atlas: between subject and atlas; Object As its name implies, refers to the name of the regions to register (e.g. brain, vertebra, tissue) (Gefen et al. 2003). In general, the image registration methods can be divided into two major groups: feature-based registration and intensity based, also called global registration (Zitová and Flusser 2003). Feature-based The feature-based registration begins with the detection of similar or equal characteristics between images. The features are distinct points from the image itself and can be significant regions (lakes, fields, organs, capillaries), lines (contours, coasts, rivers) or points (corners, intersecting lines) (Zitová and Flusser 2003). The process of feature detection does not guarantee the existence of characteristic points in all interest areas. Noise problems, occlusion of objects in an image (by motion, lighting, staining variation) are factors that affect the features matching and do not guarantee the image features have a correspondence (Zitová and Flusser 2003). This type of registration follows the next four steps (Figure 4.12):  Feature Detection: Salient and distinctive objects (closed regions, corners, edges, intersections of lines) manually or automatically detected. These characteristics are called "Control Points" (CP) (Zitová and Flusser 2003);  Features Matching: It is established the correspondence between the reference image and the image to register. Various features descriptors and similarity measures are used for the purpose (Zitová and Flusser 2003);  Model Transformation Estimation: The type and mapping function parameters which register the two images are estimated, using the correspondent points (Zitová and Flusser 2003); CHAPTER 4 – Image Preprocessing, Segmentation and Registration 51  Interpolation and Transformation: Transformation of the image according to the estimated parameters in the previous step. The non-integer coordinates are calculated using suitable interpolation techniques (Zitová and Flusser 2003). Figure 4.12: Representation of a Feature-based registration: (A) Feature Detection; (B) Feature Matching; (C) Model estimation and Transformation (adapted from Zitová and Flusser 2003). Intensity-based The Intensity-based methods preferably used when images may not present prominent details and the information is provided by the gray levels instead of local structures or forms. The intensity-based registration uses directly the intensity values of the all image, and for this reason, it is also called global registration (Zitová and Flusser 2003). This type of registration tries to find the geometric transformation that when applied to the moving image, minimizes the function cost (similarity measure). This function is computed in the overlapping regions of the input image, and the optimizer has the job of defining the search Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 52 strategy. The interpolator idea is to resample the voxel intensity into the new coordinate system according to the geometric transformation found (Figure 4.13) (Oliveira and Tavares 2012). Figure 4.13: Typical process of an intensity-based registration (from Oliveira and Tavares 2012). If possible, a pre-alignment can improve the intensity-based registration process since it can reduce the divergence of the algorithm (Oliveira and Tavares 2012). Likewise, the registration accuracy is dependent of the motion presence, lighting and staining variation. Another disadvantage of intensity-based methods is that they are computationally heavy. Similarity Measures The similarity measure is a method for evaluate and quantify the similarity between two images. The most commonly used similarity measures are based on intensity differences, intensity crosscorrelation and information theory (Oliveira and Tavares 2012). The Sum of Square Differences (SSD), also called “least squares” is a similarity measure based on intensity differences (Oliveira 2009). This metric assumes that the correspondent structures in both images have the same intensities. The smaller the value SSD, the greater is the similarity between the images (Oliveira and Tavares 2012). The SSD is given by: QQR = G ( % J − . J )  S J P A (4.8) CHAPTER 4 – Image Preprocessing, Segmentation and Registration 53 where T represents the total pixels number at the consider domain, % J is the intensity of the reference image at the pixel i and . J is the intensity of the image to align at the pixel +. (Oliveira 2009; Oliveira and Tavares 2011). Generally, it is used the normalized SSD, also called MSE. Its formula was the form: UQV = 1 T G ( % J − . J )  S J P A (4.9) The SSD measures are very sensitive to pixels with a high intensity difference between them, for example, due to noise (Oliveira 2009). The cross-correlation (CC), as the name suggests, is similarity measure based on intensity crosscorrelation. Its assumption relies on the existence of a linear relation between the intensities of the corresponding structures in both images (Oliveira and Tavares 2012). The CC is given by: @@ = ∑ : % J − % X < ( . J − . X ) S J P A Y ∑ : % J − % X <  × ∑ ( . J − . X )  S J P A S J P A (4.10) where % J ,. J and N are the same parameters defined by SSD and %X and .X are the intensity mean of the images f and g at the consider domain (Oliveira and Tavares 2011). The CC varies between -1 (minus one) and 1 (one). If CC=1, the images are identical; if CC=0, the images does not have resemblances and if CC= -1 there is a strong linear dependence, e.g. when the pixel intensity of an image increase and the pixel intensity of the other image tends to decrease (Oliveira 2009). Both SSD and CC similarity measures are more adequate for monomodal image registration (Oliveira and Tavares 2012). The information theory measures are mostly based on the mutual information (MI). The MI is a measure that quantifies the quality how one image can explain the other image, reaching its maxima when the images are correctly registered (Oliveira 2009; Oliveira and Tavares 2012). The MI is given by: Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 60 5.1 Introduction With the aim of segment and reconstruct tissues in images, this chapter describes the global methodology to fulfill the goal of this Dissertation as well as the experimental data used to validate it. Before dealing with the methodology steps used to the purpose in mind, it becomes important to analyze in detail the experimental cases used in this work. The laboratories of the Institute of Biomedical Sciences Abel Salazar cordially offered the study material for this work. Since the acquisition of the histological images depended on the accuracy of the equipment used, a brief description of the scanner and microtome employed (laboratory equipment) was founded appropriate. The set of study cases analyzed is composed by cases with normal tissue and cases with the presence of abnormal tissue. The work methodology followed in this work took two approaches: a first approach to find a suitable geometric transformation for applying to the type of images, and a second approach with the objective of create a 3D histological volume. Preprocessing, segmentation, registration and reconstruction techniques are described. Along this chapter, the approaches and their steps will be explained in detail. The development and testing of the different image tasks were made in Matlab2012b® (Inc., Natick, Massachusetts, United States) and some parts of the main Matlab codes are presented. The quantitative methods applied to assess accuracy of the histological alignment are described. 5.2 Study Cases Various health institutions (hospitals, health centers, among others) rely on specialized laboratories for detailed analysis of tissues. The laboratory process for obtaining histological images, starting from a chosen tissue fragment, was performed in the Pathology and Cytology laboratory of the Biomedical Sciences Abel Salazar Institute. Four tissue datasets were collected and analyzed in this Dissertation. In order to obtain the digital images, it was firstly necessary to study the Olympus program (scanner program) and transform the output images (.vsi) to a recognized format to work with Matlab2012b. 5.2.1 Materials and Methods After selected a tissue piece, was carried out its preparation in accordance with the procedures de scribed in detail in chapter 2 fragment to facilitate the subsequent image registration (F followed the normal protocol. Table 5.1 summarizes the procedures, time and tools used to obtain the images cases to be used. It should be mentioned that prior to scanning, was made a final cleaning of the individual blades, as they contained residues of agent materials, some superficial dust and some blades presenting dislocation of the lamella to the boundaries were red into account due to the sensitivity of the image acquisition by Olympus and subsequent image quality. Since the acquisition of the histological images used, a brief description of the scanner (Olympus VS110) an employed was founded appropriate Figure 5.1: Example of the p CHAPTER 5 – Data a nd Methods Implemented 61 Materials and Methods After selected a tissue piece, was carried out its preparation in accordance with the procedures scribed in detail in chapter 2 . Four markers were added to the init ial case fragment to facilitate the subsequent image registration (F igure 5.1). The three posterior cases followed the normal protocol. summarizes the procedures, time and tools used to obtain the images It should be mentioned that prior to scanning, was made a final cleaning of the individual blades, as they contained residues of agent materials, some superficial dust and some blades presenting dislocation of the lamella to the boundaries were red one. These were factors taken into account due to the sensitivity of the image acquisition by Olympus and subsequent image histological images was depended on the accuracy of the scanner (Olympus VS110) an d microtome (Leica RM 2255) appropriate . p araffin block (left) and tissue slide numbered (right) nd Methods Implemented After selected a tissue piece, was carried out its preparation in accordance with the procedures ial case around the tissue The three posterior cases summarizes the procedures, time and tools used to obtain the images for the four It should be mentioned that prior to scanning, was made a final cleaning of the individual blades, as they contained residues of agent materials, some superficial dust and some blades one. These were factors taken into account due to the sensitivity of the image acquisition by Olympus and subsequent image the accuracy of the equipment d microtome (Leica RM 2255) numbered (right) (initial case). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 62 Table 5.1: Laboratory Tissue Procedures. Materials and time. DataSet Description Tissue Procedures Digitalization Total Time Total Images Fixation Inclusion Cutting and Stainning Scanner: Olympus VS110 Magnification: 20x Agent: Formalin Agent: Paraffin Stain: HE (Hematoxylin and eosin) 124 INITIAL CASE Testicular Tumoral Tissue (dog) (2 days) (1 day) (± 7 days) ±10 days 100 CASE 1 Normal lymph node (dog) (1,5 day) (1 day) (± 5 days) ± 7 days 100 CASE 2 Lymph node tumor (dog) (1,5 day) (1 day) (± 5 days) ± 7 days 96 CASE 3 Mammary gland tumor (dog) (2 days) (1 day) (± 6 days) ±9 days Scanner The scanner used to obtain the histological images was the Olympus VS110 - Digital virtual microscopy system (Figure 5.2) by Olympus America Inc. Its software allows to the user a full control over the details of the scanning process. The VS110 system is based on an upright motorized Olympus microscope presenting four Olympus PlanSApo objectives - 2x, 10x, 20x and 40x. In standard configuration this scanner enables allows automated tissue detection. The VS110 system is capable of scanning even large specimens in multiple z-planes (Instruments 2013; OlympusAmerica 2010). Some specifications of the Olympus VS110 are: Width: 720 mm; Depth: 598 mm; Height: 638 mm; Weight: 100 kg; Power consumption: 1,024 W Resolution 0.32 µm/pixel (for objective 20x at Figure 5.2: Olympus VS110 - To acquire the digital images, it was necessary to learn the functioning of the scanner software. Therefore, was made use of the device manual and t laboratory. Because the device was acquired by the Pathology laboratory facilities for study and research, permission had to be guaranteed. Also, the sensibility of the device lead to problems in digitalizing all the ima took more time than the necessary. It is important to mention that the digitalization process was carried out according to the laboratory schedule. Microtome The device used for cutting the tissue This equipment allows motorized and manual sectioning for producing high comes with stepping motor, disposable blade holder, and quick release specimen clamp and foot pedal (Leicabiosystems 2013 ; Some of its characteristics of the Width: 413 mm; Depth: 618 mm; Height: 305 mm; Cassette/block of paraffin dimensions: 50 x 60 x 40 mm Weight: 37 kg (without accessories) CHAPTER 5 – Data a nd Methods Implemented 63 Power consumption: 1,024 W ; m/pixel (for objective 20x at standard scanning) ( OlympusAmerica 201 - Digital virtual microscopy system (adapted from Olympus 2013) digital images, it was necessary to learn the functioning of the scanner software. Therefore, was made use of the device manual and t echnical assistance provided by the laboratory. Because the device was acquired by the Pathology laboratory facilities for study and research, permission had to be guaranteed. Also, the sensibility of the device lead to problems in digitalizing all the ima ges and the process took more time than the necessary. It is important to mention that the digitalization process was carried out according to the The device used for cutting the tissue -paraffin block was used the Leica RM 2255 This equipment allows motorized and manual sectioning for producing high - quality sections. It comes with stepping motor, disposable blade holder, and quick release specimen clamp and foot ; Microsystems 2011). of the Leica RM 2255 are: Cassette/block of paraffin dimensions: 50 x 60 x 40 mm ; Weight: 37 kg (without accessories) (Microsystems 2011). nd Methods Implemented OlympusAmerica 201 0). Olympus 2013) . digital images, it was necessary to learn the functioning of the scanner software. echnical assistance provided by the laboratory. Because the device was acquired by the Pathology laboratory facilities for study and ges and the process It is important to mention that the digitalization process was carried out according to the Leica RM 2255 (Figure 5.3). quality sections. It comes with stepping motor, disposable blade holder, and quick release specimen clamp and foot Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 64 As this device requires an extra care and some experiential technique to perform accurate cuts, this step was performed by an expert. In all the cases, the cuts were performed with a 3 µm (micrometers) of thickness. Figure 5.3: Leica RM2255 device (from Leicabiosystems 2013) 5.2.2 Tissue Analysis For all of the analyzed cases in this Dissertation, the tissues are derived from the domestic dog, also called Canis lupus familiaris. After consulting books of histological domain, namely Bacha and Wood (1990), and tissue analysis by a light microscopy it was possible to make some conclusions. Further analysis by a histologist confirmed the suspicions. The initial case, besides presenting several representative structures of the reproductive system, presents a possible tumor. A microscopic analysis showed Leydig and Sertoli abnormal cells presence, clarifying the existence of a collision tumor (two different cell lines). Figure 5.4 identifies the principal components present in the various histological images of the initial case. The case 1 presents a normal lymph node. The lymph nodes are an oval-shaped organ of the immune system, distributed widely throughout the body. They act as filters of foreign particles to the body and are important in the proper functioning of the immune system (Bacha and Wood 1990). The lymphatic cells are normal and there is no presence of tumor cells, can be seen in Figure 5.5. On the other hand, the case 2 presents a lymph metastasis (Figure 5.6). Cancer cells can acquire the ability to penetrate the walls of lymphatic and blood vessels. In this way, circulating tumor cells can go to other sites and tissues in the body. Only malignant tumor cells have this capacity CHAPTER 5 – Data and Methods Implemented 65 to metastasize (Bacha and Wood 1990). This type of tumor is only perceived at a microscopic scale. Finally, the case 3 presents a mammary gland tumor. Fat tissue and skin hair surround the gland. Muscular tissue responsible for contraction of the skin hair and breastfeeding functions is also present. A massive tumor is immediately noted for the color and textured appearance (Figure 5.7). Figure 5.5: Normal lymph node analysis: 1) blood vessel; 2) Cortex and 3) Capsule (case 1). Figure 5.4: Testicular Tumoral Tissue analysis: 1) Epididymis; 2) Tumor (homogeneous mass); 3) Seminiferous tubules and 4) Connective tissue (initial case). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 66 Figure 5.7: Mammary gland tumor analysis: 1) Muscular Tissue; 2) Fat tissue and skin hair and 3) Tumor (case 3). Figure 5.6: Lymph node tumor analysis: 1) Metastasis and 2) Blood vessel (case 2). CHAPTER 5 – Data and Methods Implemented 67 5.2.3 Image Dataset With the observation of the final images, it was possible to observe several artifacts resulting from the manual process of tissue preparation: Damaged sections: During sectioning and mounting, sections are occasionally torn and folded (Figure 5.8, Figure 5.9); Orientation differences: Sections were placed in different orientations on glass slides due to the manual nature of the process (Figure 5.10A, Figure 5.10B); Luminance gradient: Sections mounted close to the edge of the glass produce images with different luminance gradients (Figure 5.10A); Non-tissue noise: Like dust and air bubbles may cause artifacts; Staining variations: Differences in staining duration and stain concentration results in color variations at the stained specimens (Figure 5.10A, Figure 5.10B) (Mosaliganti et al. 2006). Figure 5.8: Example of a damage section (case 1). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 68 Figure 5.9: Example of a damage section (case 3). For all of the four cases, these types of artifacts were taken into account before starting any task of image analyses. After reviewing carefully the resulting images for the cases, some slides were eliminated. Table 5.2 shows details on the resulting images dataset. Table 5.2: Total dataset of the cases. Dataset Total Images Excluded slices Final total Images INITIAL CASE 124 - 124 CASE 1 100 5 95 CASE 2 100 - 100 CASE 3 96 1 95 Before any morphological operation, the images from each dataset were renumbered. To do so, a small algorithm for renumbering was developed. CHAPTER 5 – Data and Methods Implemented 69 Each image is a RGB color image. The Figure 5.11 presents the red, green and blue components of a histological image from the initial case. Since the other cases were stained with the same Stain (HE), the grayscale components of the other cases’ images are similar. (A) (B) Figure 5.10: (A) slice 30 from initial case (different luminance gradients at the bottom); (B) slice 8 from initial case (different luminance and presence of staining variations). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 76 function. As a result, the two images are aligned with each other (Figure 5.18) (Mathworks 2013f). In sum: where the input image is the image to align, input_points and base_points are the selected correspondent points, mytform is the geometric transformation and registered is the new image input (Mathworks 2013f). base= imread ('hist.jpg'); input= imread(‘hist2.jpg'); cpselect(input, base); mytform = cp2tform(input_points, base_points, 'projective'); registered = imtransform(input, mytform)(adapted from Mathworks 2013f). Figure 5.17: Graphical Interface with selected correspondent points (GUI). CHAPTER 5 – Data and Methods Implemented 77 The number of points selected to perform the control points registration takes into account the minimum points number required to apply a given geometric transformation (Table 4.2). To compare the different alignments, in order to find the most suitable transformations to the histological images, MSE (mean square error), PSNR, and also MAXERR and L2RAT were calculated. At this point, two comparisons were established: the first, compare registration errors of the different transforms with the same points number and the second comparison, verify at the same geometric transformation, if the increasing number of control points (CP) may decrease the registration error. In order to calculate these errors, was made use of the measerr function. This function returns approximation quality metrics, like peak signal-to-noise ratio, PSNR, mean square error, MSE, maximum squared error, MAXERR, and ratio of squared norms, L2RAT (Mathworks 2013g). The function was the form: [PSNR,MSE,MAXERR,L2RAT] = measerr(I,I´) (adapted from Mathworks 2013g) where I is the base image and the I’ is the final input image after registration. The PSNR or "peak signal-to-noise ratio" is a quality measure, in decibels, between two images. The higher the PSNR, the better is the quality. After performed this formula, the information summarized in Table 6.1 and Table 6.2 was obtained. Figure 5.18: Registration of two images using the similarity model for CP=8. Overlay of the two images where the base image is more transparent. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 78 5.3.2 Second Approach With the aiming of segment and reconstruct a 3D histological volume, was developed a methodology with four steps: Preprocessing, Segmentation, Registration and Reconstruction. The first step, Preprocessing, addresses one of the most commonly used filters for histological image smoothing: the Gaussian filter. In the Segmentation step are mentioned three methods to extract the tissue from the background: an Otsu thresholding is applied to a grey level image, an Otsu thresholding is applied to the saturation component of HSV and the application of a color-based segmentation using k-means clustering. The step three, Registration, aimed to test and compare registration techniques (even their combination) such as: (A) Rotation and Scale of an Image Using Automated Feature Matching (automatic feature-based) and (B) Intensity-Based Automatic Image Registration (intensity-based) proposed by Matlab2012b. Also, a registration model was analyzed (slice registration with only the reference image and slice registration pairwise). At last, the Reconstruction step made use of techniques for visualizing the scalar data, obtained in the previous step. Image Preprocessing Since the histological images are digital images, they are prone to various types of noise; result of the acquisition method. Thus, methods like image smoothing, denoising and enhancement may be applied for histological image restoration (He et al. 2012). Image smoothing usually refers to spatial filtering to highlight the major image structure by removing image noise and fine details, such as Gaussian filtering (Alves 2013; He et al. 2012). The application of a Gaussian filter to the image was accomplished using the Matlab function fspecial. This function returns a rotationally symmetric Gaussian lowpass filter of size hsize with standard deviation sigma (positive). hsize can be a vector specifying the number of rows and columns in h, or it can be a scalar, in which case h is a square matrix (Gonzalez et al. 2004). The code used for this purpose has the following form: CHAPTER 5 – Data and Methods Implemented 79 I = imread ('hist.jpg'); h = fspecial('gaussian', hsize, sigma); w=imfilter(I,h); imshow (w) (adapted from Mathworks 2013h) where I is the image, h is the Gaussian filter and w is the smoothed image (Mathworks 2013h). The results are presented in Figure 6.3. Image Segmentation For the tissue reconstructing it was necessary to use segmentation techniques. The segmentation in this study aimed to separate the tissue (foreground) from the background. Unlike radiology images, histology images usually have background pixels with high luminance values. During registration, the high background luminance may significantly affect the metric value and decrease registration accuracy (Mosaliganti et al. 2006). To find the suitable segmentation technique for the study cases, several methods are studied including: Global thresholding using Grayscale image, Global thresholding using HSV color space and Color-Based Segmentation using K-Means Clustering (Figure 5.19). Figure 5.19: Segmentation step of the computational pipeline developed. Segmentation Step 1: Segmentation (Comparison of techniques) Otsu Thresholding using Grayscale image Otsu Thresholding using HSV color space Color-Based Segmentation using K-Means Clustering Step 2: Post processing Cleaning and Sizing Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 80 These techniques were designed to segment the tissue and avoid luminance variations of the background. To do so, the idea of making a black background and a white object (tissue) emerged. The images used for the next process (Registration) are the multiplication result of the image with the mask created. It was also necessary proceeding to a cleaning step (in order to remove loose tissue-noise) and create a method to turn the images to the same size (necessary for the registration process). Otsu Thresholding The Matlab toolbox provides a function called graythresh that computes an automatic threshold using Otsu’s method (Otsu 1979). The sequence of steps applied for the realization of the global threshold using Matlab was: I = imread ('hist.jpg'); level = graythresh (I); BW = im2bw (I,level); imshow (BW) (adapted from Mathworks 2013i) where I is the image, level is the threshold value and BW is the image (white and black) result of thresholding with level (Mathworks 2013i). In order to separate the tissue (white) from background (black) was applied the Otsu’s method for two types of images: the first image with grayscale intensities and the second with the grayscale component of the HSV color space (saturation). Grayscale image In order to segment the input images using an automatic global threshold method, was first applied the function rgb2gray presented in Matlab toolbox to the image. For this purpose, the code used has the following form: I = imread ('hist.jpg'); CHAPTER 5 – Data and Methods Implemented 81 II=rgb2gray(I); imshow (II) (adapted from Mathworks 2013j) where I is the image and II is the gray scale image. This function creates a grayscale image from the original RGB image. This algorithm converts RGB values to grayscale values by forming a weighted sum of the R, G, and B components (Mathworks 2013j). At the end, the Otsu method was used. HSV color space The original image was transformed to the HSV color space using the rgb2hsv Matlab function and its components, namely the saturation, are separated. The followed steps are written below. I = imread ('hist.jpg'); HSV = rgb2hsv(I); S=HSV(:,:,2); imshow (S) (adapted from Mathworks 2013k) where I is the image, HSV is the respective hsv image and S is the saturation component (Mathworks 2013k). The saturation was chosen because this component focalize the tissue presence or not. At the end Otsu method was used. Color-Based Segmentation using K-Means Clustering To perform a color segmentation it was applied an example proposed by Matlab (Statistics Toolbox) using K-means clustering to images stained with hematoxylin and eosin (RGB images). This algorithm follows the following steps: 1. Convert Image from RGB Color Space to L*a*b* Color Space; To convert the RGB image to L*a*b* color space makecform and applycform functions were used. As previously referred, the L*a*b* Color Space consists of a luminosity layer 'L*', Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 82 chromaticity-layer 'a*'(indicating where color falls along the red-green axis) and chromaticitylayer 'b*' (indicating where the color falls along the blue-yellow axis). The color information is contained in the 'a*' and 'b*' layers (Mathworks 2013l). 2. Classify the Colors in 'a*b*' Space Using K-Means Clustering As seen before, clustering is a way to separate groups of objects and K-means clustering treats each object as having a location in space. K-means clustering requires that the specification of the number of clusters to be partitioned and a distance metric to quantify the closeness between two objects. Since the color information exists in the 'a*b*' space, the objects are pixels with 'a*' and 'b*' values. The number of group points (clusters) and the distance metric to measure the difference between color pixels were specified (Mathworks 2013l). 3. Label Every Pixel in the Image Using the Results from K-Means After, K-means returns an index corresponding to a cluster (cluster_index), every pixel in the image is labeled with its cluster_index (Mathworks 2013l). 4. Creating Images that Segment the H&E Image by Color Using pixel_labels, is possible separate the objects specified (number of clusters) by color. Using im2bw to one of these resulting images it was possible to obtain a black and white image (Mathworks 2013l). Segmentation Preprocessing: Cleaning and Sizing After obtaining the masks, they presented as traces of tissue loose pieces. In order to reconstruct a volume with no loose pieces it was decided to use the Matlab function called bwareopen. This function removes small objects from a binary image. Since the mask can be considered a binary image, the function was applied with the form: I = imread ('mask.jpg'); CHAPTER 5 – Data and Methods Implemented 83 Mask_final= bwareaopen(I,P) (adapted from.Mathworks 2013m) This function removes from an image I all connected components that have fewer than P white pixels (Mathworks 2013m). For the initial case, the markers are also removed for better visualization of the reconstruction. On the other hand, some slices did not present all the markers and there was the possibility of them suffering dragging. Consecutively, the RGB images were transform grayscale images (using the function rgb2gray previous mentioned) and at last, a simple multiplication operation was done between each mask and the respective gray image. This operation was done for all the images for the 4 cases. Since the registration demands that the histological images have the same dimensions (without changing intensities) a simple idea was put into practice. To all the images of the cases, was determined the maximum width and height supported by the images. Founded the maxima height and width has added to each mask, background information (black pixels) to reach the final dimensions estimated above. This ensured that all images had the same dimensions and the key information (tissue) is not affected. For this purpose, we used the function padarray which pads an array (e.g. Image) with 0's (zeros). For this purpose, the code used has the following form: I = imread ('hist.jpg'); II=padarray(I,[extra_h extra_w]); (adapted from Mathworks, 2013n) where I is the image and extra_h and extra_w are the dimensions of height and width to fill around the entire image (Mathworks, 2013n). Image Registration In this step, two different registration techniques were tested: (A) Rotation and Scale of an Image Using Automated Feature Matching (automatic feature-based) and (B) Intensity-Based Automatic Image Registration (intensity-based) proposed by Matlab2012b. The feature-based technique was implemented using the Computer Vision System Toolbox™ and the intensitybased registration was achieved using Image Processing Toolbox™ from Matlab. The principal objective focused on finding the suitable registration technique for the reconstruction of the histological cases (Figure 5.20). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images Feature-Based The technique (A) consisted in finding distorted image to the original of this technique to the hi stological images required three 1. Find Matching Features Between Images To detect similar features both images algorithm (Bay et al. 2008) was For extracting these interest points, was used also known as descriptors. The function derives the descriptors from pixels surrounding an interest point. These pixels describe and match features s (Mathworks 2013o). Finally , the features were matched by (Mathworks, 2013p). 2. Estimate the Transformation The transformation matrix is obtained by the matching point pairs using the statistically robust RANdom SAmpling Consensus (RANSAC) algorithm algorithm removes outliers ( dataset distribution line (Mathworks 2013q) Figure 5. 20 Feature registration Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 84 consisted in finding rotation angle and scale factor to best approximate a distorted image to the original , using automatic feature detection and matching. The application stological images required three steps: Find Matching Features Between Images To detect similar features both images , was used the SpeededUp Robust Features (SURF) was used to find blob features in grayscale image interest points, was used extractFeatures function to extract The function derives the descriptors from pixels surrounding an interest point. These pixels describe and match features s peci fied by a single , the features were matched by the previous descriptors, and their location is Transformation transformation matrix is obtained by the matching point pairs using the statistically robust RANdom SAmpling Consensus (RANSAC) algorithm (Bolles and Fischeler ( Figure 5.21B), which are points that do not fit into a (Mathworks 2013q) . 20 : Methodology adopted in the Registration step. Registration techniques tested: Feature -based registration Intensity-based registration Segmentation and 3D Reconstruction of Animal Tissues in Histological Images to best approximate a , using automatic feature detection and matching. The application Up Robust Features (SURF) in grayscale image s (Figure 5.21A). function to extract feature vectors, The function derives the descriptors from pixels surrounding an fied by a single -point location their location is kept transformation matrix is obtained by the matching point pairs using the statistically robust and Fischeler 1981). This do not fit into a n optimal CHAPTER 5 – Data and Methods Implemented 85 3. Recover the original images Finally the distorted image is recovered, applying the transformation matrix to the image (Figure 5.21C) (Mathworks, 2013p). After applying this algorithm to all images, certain parameter adjustments were performed using some images. Speeded-Up Robust Features (SURF) SURF (Speeded-Up Robust Features) is a novel scaleand rotation-invariant detector and descriptor that offers a good compromise between feature complexity and robustness to commonly occurring deformations (Bay et al. 2008). The strategy applied by SURF algorithm can be divided into three steps: • Point Detection It detector is based on the Hessian matrix because of its good performance in accuracy, and the blob-like structures are detected at locations where the hessian matrix determinant is maximum. The scale selection is also determined by the determinant of Hessian (Bay et al. 2008): [ (  ,  ) = d >  (  ,  ) >  (  ,  ) >  (  ,  ) >  (  ,  ) e (5.4) where >  (,) is the convolution of the Gaussian second order derivate f  f  .() with the image I in point x, and similary for >  (,) and >  (,) (Bay et al. 2008). • Point Description In order to be invariant to image rotation, the descriptor used by SURF is based on sums of Haar wavelet components. For that purpose, the Haar wavelet responses in x and y directions are calculated within a circular neighborhood around the interest point, with s the scale at which the interest point was detected (Bay et al. 2008). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 92 Figure 5.26: Reconstruction using a Feature-Based reconstruction with Affine transformation (initial case). Registration Accuracy To evaluate the registration precision is not enough just to assess the visual/smoothness of the reconstruction (more smooth and uniform or gruff and misshapen). Quantitative methods were used to assess the effectiveness of the alignment. To this end, three methods were used: Root Mean Square Error (RMSE) calculation (points location): Automatically determined feature points (by Surf algorithm) were used to assess the alignment error. Through the matched features, it was possible to determine the expected and the observed position obtaining the root mean square error (RMSE) for both registration types, the featurebased and intensity-based registrations. It was necessary to calculate the mean for the errors of each corresponding point among two images (these being already aligned), and then the average CHAPTER 5 – Data and Methods Implemented 93 error between all the images of the registered volume, given the mean square error (MSE). In the end, was obtained the RMSE value, applying the square root to this global MSE value. The RMSE (Root Mean Square Error) varies of the MSE and is also used for calculating the registration error, even in histological images (Caldas-Magalhaes et al, 2012; Egger et al. 2012; Sharma and Katz 2011; Sharma et al. 2011). Root Mean Square Error (RMSE) calculation (intensity): The intensities of the pixel images, were directly used for calculate the MSE (Equation 4.9) and further the RMSE, between the areas occupied by the tissues (after the registration between two images). A global value from RMSE was obtained for all the aligned images within a volume. Dice Similarity coefficient (DSC or DICE) calculation: The calculation of the Dice Similarity Coefficient (DSC) is based on the overlap of the aligned areas and it is a usual method to evaluate the value of intersection between two areas. If the two areas are equal, the DSC is a scalar between 0 and 1, with higher values representing better quality registration. (Alterovitz et al. 2006; Klein et al. 2009). This method is widely used to access the histological registration error (Alterovitz et al. 2006; Klein et al. 2009). Because the anatomical body parts sliced in pieces may have different areas, the DSC gives two informations simultaneously, one is the relative overlapping between the two areas, and the second information is the area difference. Considering two slices (slice A and slice B), if the area of slice A is minor than slice B then, after the registration, the area of slice A may be contained in the area of slice B (100% of intersection) but the DSC is not going to be 1. For example, if A=0.2 and B=0.8 and the intersection area (R) is equal to the smallest area A (R=0.2), the DSC result is 0.4., as described by the following equation: 2 a a + g = RQ@ (5.5) In order to oblige the Dice value change between 0 (0% intersection) and 1 (100% intersection), in this work it was proposed a normalization of the DSC. The original DSC value is: 2 h a + g =  (5.6) Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 94 Applying a rule of three making the equation 5.5 equal to equation5.6 we obtain,  = 2 h a + g 2 a a + g ⇔ 2 h ( a + g ) 2 a ( a + g ) ⇔ h a (5.7) The normalized DSC (DICE n ) will be: R+3 b = h j+4&  66 (5.8) where R is the area of intersection between two areas and minor area, as the name suggests, is the smallest area between these two areas. This way, the normalized DSC may change between 0 and 1 with different areas of the anatomical body parts sliced. 5.4 Summary In this chapter, the entire work methodology adopted in this project was shortly described. The methodology was divided into two approaches. The first approach had the goal of finding a suitable geometric transformation for applying to the histological images and it was analyzed a Control Points Registration. To perform this registration, correspondent points were selected in the images and was applied a specific transformation. Since the histological images had different sizes due to manual cut through the scanner interface, it was necessary to develop, previously, a methodology to resize the images (to the same dimension but keeping the intensities) - the window of cut algorithm. In the second approach, the Preprocessing step addressed the Gaussian filter image smoothing, widely used in histological images. At the segmentation step, three methods are mentioned (Otsu Threshold using the Grayscale image, Otsu Thresholding using the saturation component of the HSV color space and Color-Based Segmentation using K-Means Clustering). On the other hand, two different registration techniques were tested: (A) Rotation and Scale of an Image Using Automated Feature Matching (automatic feature-based) and (B) Intensity-Based Automatic Image Registration (intensity-based). It was considered important to develop a comparison between a registration by reference slice and by pairwise. The measures RMSE by points location, RMSE by intensities, Dice coefficient and the improved Dice (normalized Dice) were suggested for assessing the histological registration accuracy. Finally, a reconstruction solution was described for representing the bests registrations obtained. 95 CHAPTER 6 – RESULTS AND DISCUSSION 6.1 Introduction 6.2 First Approach 6.2.1 Window of Cut algorithm 6.2.2 Control Points Registration 6.3 Second Approach 6.3.1 Image Preprocessing 6.3.2 Image Segmentation 6.3.3 Image Registration 6.3.4 Reconstruction 6.4 Summary Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 96 6.1 Introduction In this section, the obtained experimental results for the two approaches mentioned in the previous chapter are presented. Firstly, the results for the window of cut and the control points registration (first approach) are described. Secondly, the results obtained in the second approach, using the selected Preprocessing, Segmentation and Registration methods (presented in the previous chapter) are compared and discussed. The 3D representation of the volumes is also presented. The optimized parameters are indicated for each process, and it is explained the reason for the choices done. 6.2 First Approach 6.2.1 Window of Cut algorithm The window cut is a method based in the centroid of a polygon whose vertices rely on the centroids. The method developed proved to be efficient in keeping the important information, that is, the tissue. This algorithm works for two or more images. Figure 6.1 and Figure 6.2 show the results of the window of cut for two slices. (A) (B) Figure 6.1: (A) Image of slice number 49 and (B) Image of slice number 50 (originals from initial case). CHAPTER 6 – Results and Discussion 97 (A) (B) Figure 6.2: (A) Image of slice number 49 and (B) Image of slice number 50, with the same size (initial case). 6.2.2 Control Points Registration As mentioned before, two comparisons were established: the first, compare registration errors of the different transforms with the same points number and the second comparison, verify at the same geometric transformation, if the increasing number of control points (CP) may decrease the registration error. The error values obtained by the application of different transforms and different points number (CP) are resumed in Table 6.1 and Table 6.2. After analyze these values, it was possible to draw certain conclusions. Considering most important the values obtained by the PSNR and MSE, it was found that the best results were obtained for the 3th order polynomial transform with sixteen control points (CP = 16). In general, the worst results were obtained with the application of polynomial models (mainly for the 4th order polynomial). Contrary to the expected for the similarity transform, the results had worse behavior for CP = 16 than when using four, eight and twelve control points. The choice of the last four control points did not benefit the results of the transform that preserves angles and curvatures. Relatively to the affine transform, the results for the sixteen points were worse comparing when used twelve control points. For the same reason, the manual selection of the last four control points did not helped minimize the error. An important point to consider is that the histological images suffer small internal distortions (swelling), result of the tissue preparation processes (Fixation, Inclusion, among others). This may be the cause of favorable results for the 3th order polynomial. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 98 In brief, the results showed two major conclusions. It must be taking into account the fact that the point’s choice was done manually and could not respond to reality, even with the help from a specialist. On the other hand, the prior knowledge of the distortion factors caused by tissue preparation procedures, explains the more favorable results for the 3th order polynomial. In addition, some factors have led to the rejection of this control points method for future reconstruction application. Apart from being a slow method, the point collecting became fatiguing with difficult correspondence (when the images could be positioned in different directions on the slide). Therefore, was decided to embark on other registration techniques using an automatic choice of points (faster). Table 6.1: Quality parameters for Similarity, Affine and Projective transformations. (*)Without registration MSE: 567.12 (*) PSNR: 20.59 (*) MAXERR: 179 (*) L2RAT: 1.05 (*) Similarity (minimum 2 points) Afine (minimum 3 points) Projetive (minimum 4 points) CP MSE PSNR MAXER R L2RAT MSE PSNR MAXER R L2RAT MSE PSNR MAXERR L2RAT 4 429.88 21.80 181 1.04 416.58 21.93 180 1.04 489.35 21.28 165 1.03 8 429.10 21.81 182 1.04 415.46 21.95 179 1.04 421.97 21.88 180 1.04 12 427.63 21.82 181 1.04 409.91 22.00 178 1.04 411.84 21.98 180 1.04 16 430.53 21.79 183 1.04 415.29 21.95 178 1.04 408.14 22.02 176 1.04 Table 6.2: Quality parameters for Polynomial of order 2, 3 and 4 transformations. Polinomial Order 2 (minimum 6 points) Polinomial Order 3 (minimum 10 points) Polinomial Order 4 (minimum 15 points) CP MSE PSNR MAXERR L2RAT MSE PSNR MAXERR L2RAT MSE PSNR MAXERR L2RAT 4 - - - - - - - - - - - - 8 427.28 21.82 180 1.03 - - - - - - - - 12 411.77 21.98 208 1.03 428.79 21.81 219 1.07 - - - - 16 373.69 22.41 175 1.04 335.61 22.87 219 1.05 590.45 20.42 198 1.02 CHAPTER 6 – Results and Discussion 99 6.3 Second Approach 6.3.1 Image Preprocessing As seen in the literature (Alves 2013) and after testing the image with various Gaussian filters, it was concluded that the higher is sigma, the greater is the image blur, not depending much of the parameter related to the windows size. Figure 6.3 shows the same image applying different sigmas () without changing the window size. In order to obtain a good smoothing, but the most expected values, keeping some general tissue details, were a Gaussian filter with size window of [3 3] and sigma equal to 4. Applying the smoothing filter to the all images is equal to apply the filter to the three image components separately. The results of the segmentation step will demonstrate the importance of this smoothing step in the tissue segmentation. 6.3.2 Image Segmentation To find the best process for segmenting the tissue the three mentioned segmentation methods were tested. The results are shown and discussed in the following. The image cleaning and sizing, were achieved with success. Otsu Threshold using Grayscale Image After creating a grayscale image starting from the color image, an Otsu threshold was performed (Figure 6.4). The value of threshold used to separate the objects (tissue and background) was determined automatically by Otsu formula. Otsu Threshold using HSV color space After separating the components of HSV color space starting from the color image, an Otsu threshold was performed on the saturation component (Figure 6.5). The value of threshold used Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 100 to separate the objects (tissue and background) was determined automatically by the Otsu formula. (A) (B) (C) Figure 6.3: Results of Gaussian filter to the original image (A) with: (B) Gaussian filter with   =  and (C) Gaussian filter with  =  (initial case). CHAPTER 6 – Results and Discussion 101 Color-Based segmentation using k-means Clustering Using this algorithm, with the same aim (separate the tissue from the background), 2 clusters were specified (one for the background and one for the tissue) and an Euclidean distance metric (default metric) was kept (since this metric was also used for segmentation of histological images with HE coloration at Mathworks 2013l). A result of this color segmentation is presented in Figure 6.6. This method proved to be efficient in the initial case, besides having a slow processing, but failed in the remaining tissue cases (Figure 6.7 and Figure 6.8). (A) (B) Figure 6.4: Result of the Otsu’s method on (A) original image with rgb2gray and (B) smoothed image with rgb2gray (slice number 60) (initial case). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 108 Figure 6.16: Grayscale Image with mask from slice 5 (case 1). Figure 6.17: Grayscale Image with mask from slice 35 (case 3). CHAPTER 6 – Results and Discussion 109 (A) (B) (C) (D) Figure 6.18: Sizing: (A) Original image from slice number 70; (B) Sized image from slice 70; (C) Original image from slice number 72 and (D) Sized image from slice 72 (case 2). The Table 6.3 shows the final dimensions, that is the length and width for all the images that compose the 4 cases. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 110 Table 6.3: Final size of the analyzed cases. Cases Final Size Initial case 1150 x 700 Case 1 400 x 200 Case 2 353 x 400 Case 3 1000 x 800 6.3.3 Image Registration At this point, the grayscale images with the same size were concatenated into a 3D matrix. Four matrixes were produced, corresponding to the four cases. In this step, the feature-based and intensity-based techniques were put to the test. For these techniques were tested the two models mentioned previously: the pairwise registration or by a reference slice. The reference slices chosen for the cases were: initial case (slice number 60), case 1(slice number 47), case 2 (slice number 50) and case 3 (slice number 51). This choice was made based on the fact that these sections were closest to the center and contained generally the most tissue. Feature-Based From the range of geometric transformed to apply, were tested the Nonreflective similarity and Affine transform. The projective transform was excluded because this transformation supports tilting and the slices were obtained in parallel to each other, and the cut was made horizontally along the cutting axis (Mathworks 2013q). Also, the control points registration using the projective transform presented worst results, reaffirming the elimination of its use. For a good algorithm performance, the parameter 'Desired confidence' was increased to 99.8, as well as the number of Maximum number of random samplings, ‘MaximumRandomSamples’, CHAPTER 6 – Results and Discussion 111 for 8000. The other parameters used coincide with the standard values of Matlab at (Mathworks, 2013p). Figure 5.21B and Figure 5.21C shows a feature-based registration performed in the initial case, with this standard values. This registration type failed in the cases 1 and 2, because the Speeded-Up Robust Features (SURF) algorithm could not find blob features in the grayscale images of the volumes. The calculation of the RMSE in the matched features (points) was also compromised. Also this RMSE method needed to be deeply studied, since after testing this method with the registered volumes, it was possible to note that the SURF algorithm detects random features and on each iteration, a different features number was detected (consequence of RANSAC selection). These two facts introduced small variations of RMSE values (based on the points location). Nevertheless, the Table 6.4 presents the registration results for the initial case, including the values of the RMSE by intensities, DICE and normalized DICE. After carry out the feature registrations, it was possible to draw some conclusions. Based on the analysis of the error (RMSE) and dices (DICE and DICE n ) values obtained, first of all, it seems to exist a relationship between these variables: the higher is the quadratic error; the lower is the dice coefficients. This is explained because if the registration was low accuracy (fairly rigorous) the RMSE value increase and the intersection area of between tissues is minor. On the other hand, comparing the RMSE values and DICE before and after feature registration, it was found that this type of technique improved the image registration. Pairwise registration had better results with respect to the registration via reference slice. Thus, the noteworthy are the affine and similarity alignments with low error values and good results for dices (the normalized dice presented higher value since it is not affected when two tissue pieces have different sizes). Table 6.4: Quality parameters for Similarity and Affine transformations with Feature-Based registration. Initial case (reference slice 60) Without registration RMSE= 9.3219 DICE=0.8292 DICE n =0.8458 Similarity Affine RMSE DICE DICE n RMSE DICE DICE n PAIRWISE 8.2106 0.9415 0.9630 8.1368 0.9418 0.9625 REF 9.0512 0.8064 0.9011 9.0830 0.7961 0.9006 Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 112 Intensity-Based A monomodal configuration was used because the images were acquired with the same device/protocol. The geometric transformations used were the rigid, affine and the similarity. The projective was not used (like in the feature-based algorithm) by the same reason ( axis (Mathworks 2013q; Mathworks 2013r). The pyramid levels number used during the registration process was specified to 3 levels. Other metrics are also optimized for the registration: A. Similarity metric: A mean square metric was used since this measure is more appropriate for monomodal registrations. The mean squares image similarity metric is computed by squaring the difference of corresponding pixels in each image and taking the mean of those squared differences. B. Optimizer: For this process was used the OnePlusOneEvolutionary object, which describes a one-plus-one evolutionary optimization configuration. An evolutionary algorithm iterates to find a set of parameters that produce the best possible registration result. It is done by perturbing, or mutating, the parameters from the last iteration. If the new (child) parameters yield a better result, then the child becomes the new parent, given opportunity for made use of more aggressively parameters. If the parent yields a better result, it remains the parent and the next perturbation is less aggressive (Mattes et al. 2001). It relies the value of variables such as GrowthFactor (a positive scalar value that control the rate at which the search radius grows), Epsilon (positive scalar value that controls the accuracy of convergence by adjusting the minimum size of the search radius), InitialRadius (positive scalar value that controls the initial search radius of the optimizer) and MaximumIterations (a positive scalar integer value that determines the maximum number of iterations the optimizer performs at any given pyramid level). CHAPTER 6 – Results and Discussion 113 The values used for these variables were the standard values of Matlab at (Mathworks 2013t) except the optimizer.MaximumIterations (optimizer.MaximumIterations = 1000). Figure 6.19 shows the importance of using a higher value of iterations. With a higher number of iterations, even with other standard values, the registration becomes better. (A) (B) Figure 6.19: Comparison of intensity registration between slices 59 and 60 with different values of iterations. With higher number of iterations like (A) iterations=500) the registration is more perfect than (B) (iterations=1000) - Rigid transformation. (initial case). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 114 The intensity registration resulted from all the 4 cases and the results of the intensity transformations are shown in Table 6.5, Table 6.6, Table 6.7 and Table 6.8. Table 6.5: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (initial case). Initial case (slice reference number 60) Without registration RMSE= 9.3219 DICE= 0.8292 DICE n = 0.8458 PreRegistration RMSE= 8.6835 DICE= 0.9441 DICE n = 0.9635 Rigid Similarity Affine RMSE DICE DICE n RMSE DICE DICE n RMSE DICE DICE n PAIRWISE 8.3255 0.9602 0.9798 8.1888 0.9676 0.9772 - - - REF 8.3307 0.9671 0.9753 8.8068 0.9377 0.9454 Table 6.6: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 1). Case 1 (slice reference number 47) Without registration RMSE= 9.5687 DICE= 0.8606 DICE n = 0.8734 PreRegistration RMSE= 7.9695 DICE= 0.9356 DICE n = 0.9517 Rigid Similarity Affine RMSE DICE DICE n RMSE DICE DICE n RMSE DICE DICE n PAIRWISE 7.4857 0.9484 0.9651 7.3995 0.9515 0.9618 9.5232 0.7744 0.7820 REF 7.5954 0.9492 0.9598 8.6138 0.9066 0.9156 Table 6.7: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 2). Case 2 (slice reference number 50) Without registration RMSE= 9.5544 DICE= 0.8222 DICE n = 0.8423 PreRegistration RMSE= 7.7236 DICE= 0.8271 DICE n = 0.8589 Rigid Similarity Affine RMSE DICE DICE n RMSE DICE DICE n RMSE DICE DICE n PAIRWISE 6.8316 0.8774 0.9121 6.8222 0.8773 0.9052 9.4064 0.5922 0.6110 REF 7.2046 0.8679 0.8927 9.0429 0.6978 0.7211 CHAPTER 6 – Results and Discussion 115 Table 6.8: Quality parameters for Similarity and Affine transformations with Intensity-Based registration (case 3). Case 3 (slice reference number 51) Without registration RMSE= 10.1506 DICE= 0.6705 DICE n = 0.7135 PreRegistration RMSE= 9.9125 DICE= 0.7698 DICE n = 0.8198 Rigid Similarity Affine RMSE DICE DICE n RMSE DICE DICE n RMSE DICE DICE n PAIRWISE 9.6133 0.8114 0.8613 9.5578 0.8191 0.8604 10.4697 0.5747 0.6061 REF 9.7034 0.8080 0.8471 10.0953 0.7273 0.7615 After carry out the intensity registrations, it was possible to draw some conclusions. Based on the analysis of the error (RMSE) and dices (DICE and DICE n ) values obtained, the same conclusion was made: the existence of a relationship between the registration accuracy methods proposed (RMSE, DICE and DICE n ) - the higher is the quadratic error; the lower is the dice coefficients. Looking for all the cases and comparing the RMSE values and DICE before and after intensity registration, it was found that this type of technique improved the image alignment. The PreRegistration step using a rigid transformation for placing the tissue in the image center, in order to facilitate the next intensity-based registrations and avoid divergences of the algorithms was successful. Minor error values and improved dices values are noted. As in the feature-based registration, the pairwise registration had better results with respect to the registration via reference slice. Globally, the best results for the intensity registration were using a pairwise model and a similarity transform. Indeed, it was observed in all cases. The worst results were observed when using an affine transformation. The RMSE values are relatively high, and the dice values are relatively low. The affine transformation performs rotation, translation, scale and shear; too many freedom degrees to the registration type needed for the histological images. The affine pairwise registration for the initial case was not achieved and when achieved, the results do not have a good appearance (Figure 6.20). In fact, when the affine registration by slice reference presented better results (in comparison with the pairwise registration) since this type of registration model does not accumulate error (as each alignment is performed). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 116 On the other hand, the rigid registration only performs rotation and translation (simple transformation), and the error values and dices presented for this registration presented even good results (Figure 6.21). It was also possible to verify that the normalize dice, DICE n , presented better and higher results than the normal dice, DICE (The same occur at the feature-based registration). A plausible explanation for this is the fact that the normalized dice is less affected between slices with different areas. Since the registration works with histological images it is normal that the anatomical body’s presented (as unique to each image) have different areas and intersection areas. For example, considering the case 3, two images with different areas are presented (Figure 6.22). Since the images are very distinct to each other, with areas completely different, the DICE presents a ‘small’ value in relation to the normalized dice value (Figure 6.23). So, in this way, it can be concluded that the normalized dice is less sensible to different areas between tissues, becoming a more accurate measure to analyze the registration (by intersection area). As mentioned above, the DICE n only depends of the intersection area and the minor area between two slices. Figure 6.20: Intensity-based registration with affine transform (case 2). CHAPTER 6 – Results and Discussion 117 Figure 6.21: Intensity-based registration with rigid pairwise transform (case 2). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 124 (A) (B) (C) Figure 6.30: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 1). CHAPTER 6 – Results and Discussion 125 (A) (B) (C) Figure 6.31: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 2). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 126 (A) (B) (C) Figure 6.32: Intensity registration based on a similarity transform at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 3). CHAPTER 6 – Results and Discussion 127 Figure 6.33: Feature registration based on a similarity transform with viewpoint: View (3) (initial case). Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 128 (A) (B) (C) Figure 6.34: Intensity registration based on a similarity transform (REFERENCE) at different views: (A) az=-35 and el=30; (B) az=70 and el=30; (C) az=140 and el=30 (case 2). CHAPTER 6 – Results and Discussion 129 6.4 Summary In the first approach, the created window of cut served its purpose and the control points registration showed that the polynomial transforms were the most suitable transforms to the image type (histological). In this order, it was possible to conclude that the feature point’s selection is a process that can influence the registration. Because the point’s choice was done manually there was a possibility of not corresponding to the reality (even with the help from a specialist). On the other hand, the prior knowledge of the distortion factors caused by tissue preparation procedures, explains the more favorable results for the elastic transforms. At the second approach, the preprocessing step with a Gaussian filter proved to be efficient. The segmentation methods studied showed that the Otsu thresholding by the saturation component achieved good results for the four analyzed cases. Speaking about the registration step, the feature-based registration failed and the intensity-based registration proved to be efficient for all the cases. The similarity transform pairwise using intensities proved to be useful for reconstructing several tissue types and to represent the best visual reconstruction (which resembles the original tissue piece) with minimum registration error. The three accuracy methods indicate to be effective to select the best reconstruction results (special mention for normalized dice). However, a quality parameter (visual reconstruction) cannot be separate from selecting a winner volume. Only these two measures linked (quantitative and qualitative) can prove the registration effectiveness. Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 130 131 CHAPTER 7 – CONCLUSIONS AND FUTURE PERSPECTIVES 7.1 Conclusions 7.2 Future Work Perspectives Segmentation and 3D Reconstruction of Animal Tissues in Histological Images 132 7.1 Conclusions Throughout this Dissertation, some conclusions were made in each step giving space to the next action. Before 3D reconstruct an object, in this case tissue pieces, it is necessary to know all about them: existent tissue types (muscular, nervous, connective and epithelial) and their characteristics. Since the histological images are the product of several techniques such as Fixation, Inclusion, Staining and instruments like the microtome, it is fundamental to also know them. It is common knowledge that the Tissue Processing introduces several types of artifacts to the images (noise, cut pleats, among others); and it is important to not mistake them with tissue self characteristics. This previous knowledge gives space to the application of Image Preprocessing, Segmentation and Registration techniques; important techniques for the accomplishment the goal, the tissue reconstruction. Along the present work, certain conclusions were withdrawn and retained, giving rise to possible future works. This Dissertation aimed the collaboration of image analysis tools with the histological area, taking the opportunity for establishing useful links between engineering and health field. After discussed several methodologies, this dissertation presented a straightforward and successfully methodology to reconstruct tissue volumes from histological images, in accordance with the established initial objectives for this Dissertation. 7.4 Future Work Perspectives This Dissertation leaves several aspects to be refined and future proposals. First of all, since the histological images are color images and its saturation depends on the factor time vs. concentration, a study of a color normalization of the images (Saraswat and Arya 2013) can be useful for the segmentation step. A future work perspective lies on the tumor tissue segmentation, if it exists in the normal tissue. Speaking about the registration step, the feature-based method can be explored, in particular, other methods to automatically select the features (rather than SURF algorithm). In this way, it will be possible to test feature registrations and combinations between feature and intensity based registrations, producing better results comparing to those achieved here. CHAPTER 7 – Conclusions and Future Perspectives 133 With the location of these features, the registration error can be assessed in order of millimeters (what was wanted to achieve). It was possible to note that the transformations with higher number of degrees of freedom (like the affine transformation) did not produce better results in the intensity registration. According to the possibility of tissues suffer dilation and retraction, it will be important to test elastic transformations based in B-splines. Thus, in future approaches, it will be done a separation of transformations that is, beginning with simple transformations (rotation, translation) and step by step reaching the most complicate ones (polynomials). For better results, the application of this elastic transforms can be made by dividing the image in small squares and then apply the transformation. In a near future, it can be used programs for reconstruct the tissue slices, like 3D slicer.