552 | Journal of Anatomy. 2022;241:552–564.wileyonlinelibrary.com/journal/joa 1 | INTRODUCTION Threedimensional (3D) computer graphics are increasingly used for communicating anatomical information and knowledge. Computerized, 3D representations of anatomy can be visualized and manipulated dynamically and thus represent a valuable tool for anatomy learning, as well as for surgical planning and simulation (Assaf & Pasternak, 2008; Fang et al., 2020; Hemminger et al., 2005; Li et al., 2017; Murgitroyd et al., 2015; Preim & Saalfeld, 2018; Soler et al., 2014; Stepan et al., 2017; Triepels et al., 2020; Yammine & Violato, 2015). For digitally rendered anatomical scenes to be visually realistic, not only geometry needs to be represented in adequate detail but ideally also color attributes of tissues and organs should be displayed as close to their natural appearance as possible. A recent systematic analysis of a range of software applications devoted to 3D visualization of human anatomy appraised the 3D models featured in current applications with an average score of 3.04 of 5 in the realism dimension (Zilverschoon et al., 2019). This was a modest score denoting that the realism level of most 3D anatomical models is largely limited to showing some detail in visual/color clarity Received: 19 October 2021 | Revised: 18 January 2022 | Accepted: 16 February 2022 DOI: 10.1111/joa.13647 ORIGINAL ARTICLE Truecolor 3D rendering of human anatomy using surfaceguided color sampling from cadaver cryosection image data: A practical approach Jon Jatsu Azkue This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2022 The Author. Journal of Anatomy published by John Wiley & Sons Ltd on behalf of Anatomical Society. Department of Neurosciences, School of Medicine and Nursery, University of the Basque Country, UPV/EHU, Leioa, Spain Correspondence Jon Jatsu Azkue, Department of Neurosciences, School of Medicine and Nursery, University of the Basque Country, UPV/EHU, Leioa, Spain. Email:
[email protected] Abstract Threedimensional computer graphics are increasingly used for scientific visualization and for communicating anatomical knowledge and data. This study presents a practical method to produce truecolor 3D surface renditions of anatomical structures. The procedure involves extracting the surface geometry of the structure of interest from a stack of cadaver cryosection images, using the extracted surface as a probe to retrieve color information from cryosection data, and mapping sampled colors back onto the surface model to produce a truecolor rendition. Organs and body parts can be rendered separately or in combination to create custom anatomical scenes. By editing the surface probe, structures of interest can be rendered as if they had been previously dissected or prepared for anatomical demonstration. The procedure is highly flexible and nondestructive, offering new opportunities to present and communicate anatomical information and knowledge in a visually realistic manner. The technical procedure is described, including freely available opensource software tools involved in the production process, and examples of color surface renderings of anatomical structures are provided. KEYWORDS 3D graphics, anatomy education, medical visualization, surface rendering
| 553 AZKUE and nonsimplistic shapes, hence rather far from the highest score corresponding to realistic models both in shape and visual details (Zilverschoon et al., 2019). The two major classes of approaches to 3D visualization, that is, surface rendering and volume rendering, process and display color differently. In volume rendering, a 2D projection is simulated by computing the absorption and emission of light rays cast through a 3D dataset composed of volumetric pixels or voxels generated from a stack of sectional image data (Drebin et al., 1988; Kaufman, 1996; Levoy, 1988). Gray scale or color data contained in voxels are usually mapped into predetermined opacity and color values using transfer functions (Drebin et al., 1988; Ney et al., 1990). Since volume renderings for medical visualization commonly target anatomical gray scale data such as those from CT and MRI scans, tissues and organs are usually represented using artificially created colors. Still, strikingly realistic renditions of the human body may still be generated even in the absence of color information in the source image data. For instance, the cinematicrendering approach produces hyperrealistic renditions out of CT imaging data by computing multiple paths of visible photons through the body tissues (Engel, 2016; Glemser et al., 2018; Paladini et al., 2015). In surface rendering, on the other hand, a structure is represented only by its surface— most commonly a polygon mesh or isosurface— and made visible by showing its appearance upon external illumination with a virtual light source. Although most implementations of surface rendering use arbitrarily assigned solid colors (Cline et al., 1987; Cook et al., 1983; Herman & Liu, 1979), vertices or faces of a polygon mesh can have associated scalar data carrying color information, and thus a surface can also be rendered with a multicolor appearance. For example, in photogrammetry, an anatomical specimen is photographed from multiple viewpoints and then a 3D point cloud of matching points is computed that reflects the object’s 3D geometry and to which color attributes from the original photographs are then mapped to mimic the object’s external visual aspect (Petriceks et al., 2018; Schenk, 2000). In addition, a 3D polygon mesh can be wrapped in a bitmap image displaying colors and textures in order to provide the rendered structure a lifelike appearance (Preim & Saalfeld, 2018; Zilverschoon et al., 2017). The use of cadaver cryosection images as source data for 3D reconstructions allows to retrieve color information of tissues and organs, providing an avenue to produce 3D renditions of any anatomical structure with its true color appearance as found in the frozen cadaver. Collections of serial, highresolution cryosection images of human cadaver specimens both of whole bodies and separate body parts have been made available, including the Visible Human Project from the National Library of Medicine (Ackerman, 1998; Ackerman et al., 2001; Spitzer et al., 1996), the Chinese Visible Human (Zhang et al., 2003, 2006), the Visible Korean Human (Kim et al., 2002; Park et al., 2005), and the Visible Ear datasets (Sørensen et al., 2002). Highresolution murine cryosection data are also available (Roy et al., 2009; Wilson et al., 2008). In the volume rendering modality, opacity transfer functions can be used to map true color properties of the tissues into opacity levels in order to manipulate the visibility of specific organs. This strategy, termed alpha blending or alpha rendering (Kahrs & Labadie, 2013), allows to highlight specific tissues based on their color properties as well as to remove unwanted components from the scene, for example, superficial tissues or the embedding gel (Gargesha et al., 2009, 2011; Kahrs & Labadie, 2013). A practical limitation of this approach is that specific transfer functions must be set in order to highlight different tissues or organs from each source dataset. In addition, tissues with similar color properties are difficult to single out based solely on opacity transfer functions. This difficulty can be overcome by circumscribing the rendered scene to subsets of the original volumetric data previously segmented from either the original cryosection image stack (Heng et al., 2006) or from registered CT scan data (Robb & Hanson, 2006). Interestingly, organs or body parts segmented from volumetric data can be surface rendered in true color by mapping color attributes from the cryosection image stack back onto the extracted surface model. This approach is advantageous in that it reduces both the storage requirements and the amount of data that has to be rendered as compared with the total number of voxels that enter into computation in volume rendering (Udupa et al., 1991). Endeavors in this direction have been reported in the scope of biomedical image visualization. Robb and Hanson (2006) provided a few examples of renditions of gross segments of the Visible Human Male abdomen after sectioning in several planes. An analogous approach was used to produce surface renderings of the anatomy of the pelvis reconstructed from the Visible Human Project dataset in the context of The Vesalius (TM) Project (Venuti et al., 2004). Despite the potentialities of realistic color rendering for anatomical visualization, the methods and tools involved in producing truecolor surface models have not yet been described or are not available in the public domain (Dai et al., 2012; Robb & Hanson, 2006; Venuti et al., 2004). The aim of this work was to demonstrate a simple and versatile procedure to create highquality, truecolor surface renditions of human anatomical structures by extracting color information from cryosection cadaver image data. Digital models created using this technique can also be rendered as if they had been previously dissected or prepared in a variety of ways for anatomical demonstration, and easily saved and combined. A set of opensource digital tools are proposed, all of which are available in the public domain, and examples of a variety of digital dissections that may be useful for presenting anatomical information and knowledge are provided. 2 | MATERIALS AND METHODS The technical procedure involved two major steps (general workflow described in Figure 1). The first was creating a surface representation of the target structure from a stack of cadaver serial cryosection images, which was accomplished using semiautomatic segmentation and the marching cubes algorithm for isosurface extraction. The second step involved retrieving color information from the subset of voxels representing the surface layer of the target structure, using the surface mesh generated in the previous step as a probe.
554 | AZKUE 2.1 | Cryosection image data Highresolution axial plane cryosection images of five cadavers from two separate collections were used. Datasets and target structures were chosen with a criterion of convenience, in order to illustrate the general properties of the method while providing examples of a variety of different looking organs and tissues. Axial sections of both the male and female cadavers from the Visible Human Project database (National Library of Medicine, Bethesda, Maryland; Ackerman, 1998; Ackerman et al., 2001; Spitzer et al., 1996) had 1 mm and 0.33 mm spacing in the zaxis, respectively, and images were 2048 × 1216 pixels, where each pixel is defined by 24 bits of color (RGB, one byte each). Cryosection images of the formalinpreserved head of a 72yearold male donor from the same database were also used (0.147 mm intervals, with image dimensions of 1056 × 1528 pixels and 24 bits of color), whose blood vessels had been filled with aralditeF (Ratiu et al., 2003). In addition, cryosection images of the Visible Head and Visible Male datasets from the Visible Korean Human collections (Kim et al., 2002; Park et al., 2005) were used (0.1 mm and 0.2 intervals and image dimensions of 4368 × 2912 and 2468 × 1407 pixels, respectively). 2.2 | Segmentation and surface postprocessing Each series of cryosection images containing a given structure of interest was cropped for the corresponding target organ using GNU Image Manipulation Program (GIMP) software, converted FIGURE 1 Flow diagram for the surfaceguided color sampling procedure. Cryosection images are converted into a volumetric data file, which can then be resliced in any orthogonal plane for segmentation. Segmentation of the submandibular gland (sb, delimited in red color) from the Visible Human Male is shown. The marching cubes algorithm generates a 3D polygon mesh representing the external surface of the segmented target object. At this stage, a surface mesh can be edited to produce a modified version of the target structure that will be rendered as having been dissected or prepared. The color sampling operation retrieves color information (shades of brown) of those voxels representing the external boundaries of the target structure, using the surface mesh outputted by the previous step as a probe (red), and then assigns color attributes to the corresponding polygon vertices. A schematic representation of this operation in a subgroup of voxels is shown. Finally, the VTK renderer creates color on polygon faces by interpolation of vertex colors and renders the surface. b: Body of mandible; c: Carotid arteries; d: Digastric muscle; f: Facial vein; h: Hyoglossus muscle; hb: Hyoid bone; j: Internal jugular vein; m: Masseter muscle; n: Deep cervical lymph nodes; p: Palatopharyngeus and pharyngeal constrictor muscles; pl: Platysma; pt: Medial pterygoid muscle; sb: Submandibular gland; sg: Styloglossus muscle; sm: Submental vein; st: Sternocleidomastoid muscle; t: Tongue
| 555 AZKUE TABLE 1 Software tools used for segmentation, color sampling, and postprocessing of surface models Subprocess Software tool Platform Availability Website Cryosection image cropping GIMP MS Windows, MacOS, GNU Linux Free, open source https://www.gimp.org/ Conversion of 2D image stacks into volumetric files Slicer 3D MS Windows, MacOS, GNU Linux Free, open source https://www.slicer.org/ 3D segmentations ITKSNAP MS Windows, MacOS, GNU Linux Free, open source http://www.itksn ap.org Edition of polygon meshes Blender MS Windows, MacOS, GNU Linux Free, open source https://www.blend er.org/ Color sampling, visualization, and file exportation The Visualization Toolkit MS Windows, MacOS, GNU Linux Free, open source https://vtk.org/ Texture baking MeshLab MS Windows, MacOS, GNU Linux Free, open source ht tps://www.meshl ab.net FIGURE 2 Implementation of the color sampling operation using VTK. Input data importation, color sampling, rendering, and file saving steps are shown, also indicating the involved filters or functions in the VTK processing pipeline. The vtkProbeFilter, core of the sampling operation, takes the source volumetric data and the polygon mesh extracted by segmentation as inputs. The filter outputs a copy of the input mesh and appends the extracted color data from scalar values to polygon vertices. A separate step converts scalar values into color for visualization. A mapper converts the polygon mesh into a virtual model for surface rendering. The resulting model can be stored as a file. A python script that performs this operation and renders the resulting surface is provided in Appendix A
556 | AZKUE into a single volumetric data file in Nearly Raw Raster Data format (NRRD; http://teem.sourc eforge.net/nrrd/) using 3D Slicer software (v. 4.10.2 r28257; https://www.slicer.org/), and then imported to ITKSNAP Medical Image Segmentation Tool software (v3.4.0; Yushkevich et al., 2006) for segmentation (Table 1). Volumetric data can then be resliced in the x- , y- , and zaxes in ITKSNAP to generate sagittal, coronal, and transverse planar views for convenient visualization during segmentation. Segmentation refers here to the process of singling out the subset of voxels within a volumetric file that represent the structure of interest. This allows to compute a 3D surface representing the external boundaries of the target structure. All segmentations were done here semiautomatically using the builtin active contour algorithm in ITKSNAP (Kass et al., 1988), as described previously (Azkue, 2021b). Active contours are computergenerated curves that propagate within the volumetric data, through a series of iterations, until they adapt to the boundaries of the target object. ITKSNAP then uses the marching cubes algorithm (Lorensen & Cline, 1987) to generate a 3D polygon mesh representing the external surface of the target object. Polygon meshes were exported as surface 3D files in Standard Tessellation Language (STL) format and then subjected to Laplacian smoothing using Blender v2.90.0 software. This procedure removes noise locally by smoothing the position of a given vertex of the mesh based on information about its immediate neighbors, while preserving the general shape of the original model (Sorkine et al., 2004). Here, 20– 35 iterations and a lambda factor of 0.2– 0.3 were used for smoothing. Additional manual smoothing was applied as needed. FIGURE 3 Successful color sampling is dependent on the accuracy of the surface probe. A surface rendition of the Visible Human Female head is shown using correctly sized (100% scale) and distorted probes. Increasing or decreasing probe size by 2%– 3% results in sampling colors from voxels outside the body (embedding blue gel) or representing subcutaneous tissues. A crosssectional image shows the relative positions of the three probes for reference
| 557 AZKUE The above process outputs a 3D polygon mesh that represents the external shape of the anatomical structure that will be ultimately rendered. In addition to the obvious possibility of rendering the target structure as a whole, a polygon mesh may be edited digitally at this stage by cutting off discrete portions of the model in order to produce what would be rendered as a dissected specimen. All such preparations were done here by applying Boolean subtraction operations using Blender. 2.3 | Surfaceguided color sampling Color information was retrieved from those voxels at positions determined by the spatial coordinates of the polygon mesh yielded by the preceding step. The sampling operation was computed here using the Visualization Toolkit (VTK, v 9.0.3), an opensource software system for 3D computer graphics and scientific visualization (Schroeder et al., 1998; https://vtk.org/). The VTK provides a variety of socalled filters or elements that receive data from other components in a visualization pipeline, modify those in a variety of ways (e.g., extract, subsample, interpolate, merge, or split input data), and then output the modified data to be handled by other elements or subprocesses. The present approach utilized the vtkProbeFilter, a tool that can retrieve scalar data contained in voxels, such as RGB data, using a polygon mesh as a probe. RGB values (1 byte each) retrieved from sampled voxels are then mapped back to the polygon mesh as color attributes. In order to produce a final surface rendition with the extracted colors, the VTK visualization pipeline assigns color to polygon faces by interpolating vertex colors. The data processing pipeline is shown in Figure 2, and a script in Python language that performs the sampling operation and renders the resulting surface is provided in Appendix A. 2.4 | Additional processing The abovedescribed procedure is sufficient to produce a 3D rendition of the target anatomical structure with its true color appearance. Often, however, saving the model with the attached color information is also desired. The X3D file format, an XMLbased format for representing 3D information developed as an improved version of the Virtual Reality Modeling Language (VRML) was used here to export produced models with the appended color attributes (Python script also provided in Appendix A). FIGURE 4 The level of detail of rendered colors and textures depends on the resolution of the source cryosection images. Reconstructions of the cervical spinal cord from the Visible Head from the Visible Korean dataset (left) and the Visible Human Male from the NLM (right) are shown, and the maximum width of the spinal cord in the corresponding original images is indicated. Dorsal and ventral horns are readily identifiable in the higher resolution model, and relatively blurred in the lower resolution one FIGURE 5 Kidney in entirety and after editing the surface probe. Surface rendition of the whole left kidney and attached ureter from the Visible Human Male are shown, as well as virtually dissected versions of the kidney, ureter, and vascular supply after sectioning in the axial and coronal planes. The morphology and internal organization of the renal pyramids are clearly discernible. Note an accessory renal artery entering the inferior pole
558 | AZKUE Optionally the model can also be subjected to texture baking or the transferring of color information to a bitmap image, which allows to handle geometry and color data separately and reduce polygon count— and thus file size— while preserving color and texture quality. The former was accomplished here using the Parametrization and Vertex Color to Texture filters in MeshLab v2020.07 software. The resulting model was exported in OBJ format, which also generated a bitmap image file in PNG format and an MTL format file linking the preceding two files. Mesh simplification was carried out using Blender. 3 | RESULTS Truecolor surface renditions shown in Figure 1 and Figures 3– 8 illustrate the main properties of the method and provide examples of a variety of different looking organs and body parts. 3.1 | General characteristics of the method The rendered structures exhibit shapes and color textures as they are found in the cadaver image dataset, and therefore they may display nonidealized colors or particularities that are usually not represented in medical illustrations, for example, color imprints made by nearby structures such as blood vessels (Figure 1) or bile impregnation of the fat surrounding the gall bladder (not shown). In addition, possible color modifications by effect of cadaver fixation or processing are also captured, such as permeation of the embedding gel (Figures 3 and 7), reddish impregnation by arterial infusion of araldite (Figure 6), or tissue abrasion during sectioning (Figure 6). The level of detail of the rendered colors and textures is directly related to the resolution of the cryosection images and thus to the amount of information available from the source dataset. An example showing surface renderings of spinal cords reconstructed FIGURE 6 The brain as a whole and after editing the surface probe. Surface renditions of the brain from the isolated head dataset in the Visible Human image collection are shown, both as a whole (top left) and following sectioning in the axial and coronal planes or exposure of the left insular cortex (bottom right) after removal of the overlying frontal, parietal, and temporal cortices. Note whitish discoloration on the posterior short gyrus and long gyrus of the insula due to abrasion during cadaver sectioning
| 559 AZKUE from cryosections of different original resolutions is provided in Figure 4. A key aspect of the technical procedure is that successful color sampling is critically dependent on accurately defining the boundaries of the target structure by the surface probe. It is shown that increasing or reducing the size of the surface probe by 2%– 3% results in retrieving color data from voxels representing the embedding blue gel or subcutaneous tissues, respectively (Figure 3). 3.2 | Capturing external and internal color features Truecolor surface renditions of a number of anatomical structures were produced, including a salivary gland (Figure 1), skin and subcutaneous tissue (Figures 3 and 7), spinal cord (Figure 4), the kidney with attached ureter and blood supply (Figure 5), the brain (Figures 6 and 7), bone (Figures 7 and 8), and muscle (Figure 8). Editing the surface probe of a given structure prior to color sampling allowed the generation of a range of versions of the model rendered as if dissected in a variety of possible ways. For example, Figure 5 shows surface renditions of the kidney exhibiting the spatial disposition of renal pyramids after cross sectioning in the axial and coronal planes. In addition to planar sectioning, a more complex dissection of the brain was simulated to expose the insular cortex by digitally removing the overlying cortical regions (Figure 6). A different editing approach is shown in Figure 7, which presents a surface rendition of the Visible Korean male after layerwise removal of soft tissues and skull bone to expose the underlying brain. 3.3 | Texture baking and polygon reduction Texture baking preserved the general visual appearance of models and allowed exportation in OBJ file format for subsequent mesh FIGURE 7 Stripping layers off the model. A surface rendition of the head of the Visible Male from the Visible Korean dataset is shown, using a surface probe from which two pieces representing portions of the skin and skull overlying the brain had been removed. Note the tenuous purple impregnation of the skin due to the embedding gel FIGURE 8 Reducing mesh complexity without losing visual realism. The right temporalis from the Visible Human Female is shown attached to the coronoid process, before texture baking (top). Two versions of the muscle model alone are shown after texture baking and reducing the vertex count by ca. 86% or by ca. 97% (bottom). The vertex count could be reduced dramatically (figures provided in Table 2) without noticeably affecting the overlying bitmap texture. The magnification boxes show the structure of the polygon mesh from approximately the same region at the various levels of simplification
560 | AZKUE simplification. As shown in Figure 8 and Table 2 using a temporalis model, mesh simplification can dramatically reduce file size while entirely preserving external appearance. 4 | DISCUSSION 4.1 | Contribution to the state of the art An advantage of volume rendering in comparison with surface rendering is direct visualization without the need for segmentation. However, volume rendering individual organs without prior segmentation is challenging, and even more so is making them appear as dissected or prepared for anatomical demonstration. On the other hand, volumerendered scenes as such cannot be saved to disk and stored for later use. The method presented here allows to produce anatomical models that can be rendered both as a whole and as if previously dissected, can be exported to disk individually, and easily combined to create custom anatomical scenes. Further, it is shown that colortextured models can be postprocessed for optimizing file size without any discernible loss of visual detail. Application of bitmap textures to surface models, including real tissue photographs, is a useful approach to mimic the natural appearance of anatomical structures (Preim & Saalfeld, 2018; Zilverschoon et al., 2017). Surface scanning and photogrammetry can be seen as refinements of this strategy, where the 3D surface structure is enriched with actual color properties from photographs of the same object. These are probably ideal approaches for digitizing prosected body parts, as shown by Petriceks et al. (2018), as well as bone specimens (Azkue, 2021a), since 3D reconstructions using such techniques require capturing the geometry of the object of interest from all possible viewpoints and consequently the scanned object needs to be detached from the body. It is noteworthy, however, that the level of detail of reconstructed models is limited by prosection quality, and there are a number of factors that can compromise the quality of models such as difficult angles, poor lighting, and occlusions (Petriceks et al., 2018). The color sampling procedure described here offers significant advantages over previous approaches. First, this is a nondestructive technique that allows to produce multiple renditions of the same anatomical structure in a variety of shapes and preparations, with the only limit of the ability to edit the surface probe digitally. Frail or collapsible structures, such as small vessels, ureters, or bile ducts, that become easily distorted or damaged during actual dissection can be reconstructed and represented accurately, and so do smaller structures difficult to access by dissection, such as the inner ear, provided that cryosection images of adequate resolution are utilized. Second, if digital 3D models are to serve as an educational resource in the context of interactive software (Murgitroyd et al., 2015; Preim & Saalfeld, 2018; Triepels et al., 2020), it is important that models of individual organs and body parts can be combined and used interactively as separate components within a scene, rather than presented as a prosected region or tissue block. Furthermore, the flexibility offered by surface 3D models can be enhanced by including sectional anatomy and registered clinical imaging data such as CT or MRI in the scene (Liu et al., 2013; MavarHaramija et al., 2015; PratsGalino et al., 2014; Robb & Hanson, 2006; Shin & Park, 2016). Surface models generated from cadaver cryosection image datasets are ideally suited in this regard since most available collections also include clinical imaging data (Ackerman, 1998; Ackerman et al., 2001; Park et al., 2005; Zhang et al., 2003, 2006). 4.2 | Representing human anatomy in a visually realistic manner Digital anatomical models are external representations of the body, where the correspondence between the representing and represented worlds is established by physical resemblance, based on aspects such as dimensionality, number of structures, spatial relationships between structures, and surface details including texture and color detail (Chan & Cheng, 2011; Palmer, 1978). Little published information is currently available as to the extent to which realism of anatomical models influences anatomy learning. Visual realism of models may be less critical for an undergraduate medical student becoming acquainted with the basic anatomy of a body region, who may also benefit from simplified, lowtechnology models (Chan, 2015; Chan & Cheng, 2011). However, a vast majority of anatomists (80%) in a recent survey agreed that using realistic models to teach anatomy is important for students to retain their anatomical knowledge (Balta et al., 2017). Continued efforts to develop TABLE 2 Metadata of a temporal muscle model subjected to texture baking and two levels of polygon reduction Vertex count File size X3D file OBJ file MTL file PNG file Originally exported 288,546 43.8 MB Prior to simplification 288,546 90.6 MB 728 bytes 53.1 MB Firstlevel simplification 38,234 5.9 MB 278 bytes 2.5 MB Secondlevel simplification 9745 1.4 MB 282 bytes 2.4 MB Notes: The surface mesh was first exported in X3D format supporting color data appended to polygon vertices. Following the texture baking operation, geometry and color data were exported in OBJ file format with associated MTL and PNG files, and subjected to polygon reduction.