Automated segmentation and classification with artificial neural networks of objects in 3D point clouds

Automated segmentation and classification with artificial neural networks of objects in 3D point clouds Approved DISSERTATION to obtain the academic degree Doktor-Ingenieur (Dr.-Ing.) submitted to the HafenCity Universität Hamburg in the field of Geodesy and Geoinformatics by Eike Ruben Barnefske Hamburg, 2023 This dissertation is simultaneously published in: Ausschuss Geodäsie der Bayerischen Akademie der Wissenschaften (DGK), Reihe C, Dissertationen, Heft Nr. 915, München 2023, ISBN 978-3-7696-5327-4, ISSN 0065-5325, www.dgk.badw.de. Submitted on: May 4, 2023 Disputation day: July 20, 2023 License Notice This work is licensed under the Creative Commons BY 4.0 License. For the scientific articles, which are part of this thesis (see appendix), licenses are valid according to the original publication. Supervisors 1. Supervisor: Prof. Dr.-Ing. Harald Sternberg HafenCity Universität Hamburg 2. Supervisor: Prof. Dr.-Ing. Alexander Reiterer Albert-Ludwigs-Universität Freiburg Additional Supervisor: Prof. Dr.-Ing. Jochen Schiewe HafenCity Universität Hamburg Danksagung Zu allererst möchte mich bei meinem Doktorvater Harald Sternberg für den stets offenen und vertrauensvollen Austausch und die Unterstützung, insbesondere, wenn es schwierig wurde, bedanken. Danke für dein Vertrauen in mich. Für die vielen guten Anmerkungen zur Forschung, die Feedbacks, die Ermutigungen, die Impulse und die Diskussionen möchte ich mich weiterhin bei Alexander Reiterer bedanken. Ein großer Dank geht an Annette Scheider, Meike Ahrend und Clemens Semmelroth, die mich nicht nur bei fachlichen Fragen, bei den durchgeführten Untersuchungen und den Korrekturen, sondern auch durch viele bestärkende und ermutigende Gespräche unterstützt haben. Bedanken möchte ich mich bei meinen Kolleginnen und Kollegen der Arbeitsgruppe und des geodätischen Labors für das gute Arbeitsumfeld, die vielen kleinen Hilfestellung, euch Offenheit und das schöne Gemeinschaftsgefühl. Ein besonderer Dank geht an meine Eltern Dieter und Gabrielle, meine Großeltern Oskar und Hella, meine Schwester Lena und meine Freundin Sarah für die unzähligen Momente, in denen ihr mich unterstützt habt, die vielen aufbauenden Gespräche und das nie nachlassende Vertrauen in mich. Bei Almut, Katharina, Lars und Jochen möchte ich mich dafür bedanken, dass ihr mich auch mal vom Schreibtisch weglocken konntet. Mit und bei euch konnte ich bei Kuchennachmittagen, auf unseren Reisen und bei den vielen tollen Events die Kraft tanken, die ich brauchte. Zuletzt möchte ich mich noch bei den Kameraden der Freiwilligen Feuerwehr Fuhlsbüttel dafür bedanken, dass ich von euch den nötigen Rückhalt, das Verständnis und die Bestärkung für das Beenden der Arbeit erfahren durfte. i Abstract The recording of objects surfaces with Light Imaging, Detection and Ranging (LIDAR) scanners is a well-established surveying method for the highly accurate and detailed geometric creation of models. The result of LIDAR recordings is a three-dimensional (3D) point cloud with geometric and spectral (intensity and color values) features that represent a geometric model of reality. This model is usually automatically extended by the human imagination with semantic information by looking at it, so that object classes, individual objects or measurement errors in the point cloud can be reliably identified. The easy interpretation of point cloud scenes and its effective recording with LIDAR scanners has led to the fact that point clouds become a quasi-format standard for 3D models, besides to mesh, voxel and parametric models. Semantic features are necessary for automatic processing of point clouds, for example, in a building information model. Currently, semantic enhancement of point cloud information is mostly done manually, and automation (e.g., via deep learning methods) is still a subject of research. In particular, Artificial Neural Networks (ANN) have proven to be effective for this task when the data and hyperparameters (HPs) are optimized. In this thesis, the PointNet ANN was used as an example to research which are optimal point cloud data and HPs. The creation of training data with manual annotation tools, the implementation and research of processes for automatic semantic segmentation, and the development of a heuristic quality model for the evaluation of point cloud datasets and of semantic segmentation processes are the central research issues. The annotation tool, Point Cloud Classification Tools (PCCT), was developed to investigate automation, training processes of annotators, and features influence. For automatic point cloud processing, influences are points from erroneous measurements, the class inequality and the semantic class definitions. Different class definitions and methods for minimizing the differences in class sizes have been developed, adaptations in point cloud pre-processing have been applied and the weighting of infrequent classes have been optimized. The research results show that optimal (data-based) HPs for semantic segmentation of a building dataset can be defined. This HP set and the approach can be used as guidelines for similar projects. An increase in recall of more than 50% for infrequently occurring classes can be achieved by algorithm-based class definition and class size consideration. Using the heuristic quality model, available training data and semantic segmentations can be evaluated and compared. iii Zusammenfassung Die flächenhafte Erfassung von Objektoberflächen mit Light imaging, detection and ranging (LIDAR) Scannern ist ein etabliertes Vermessungsverfahren zur hoch-genauen und detailreichen geometrischen Erstellung von Modellen. Das Ergebnis der LIDAR Erfassung ist eine dreidimensionale (3D) Punktwolke mit geometrischen und spektralen Merkmalen, die ein geometrisches Modell der Realität darstellen. Dieses Modell kann durch Menschen beim Betrachten meist automatisch um semantische Informationen erweitert werden, so dass Objektklassen, einzelne Objekte oder Messfehler in der Punktwolke sicher erkannt werden. Die einfache Interpretation durch den Menschen von Punktwolkenszenen und deren effektiven Erfassung mit LIDAR Scannern hat dazu geführt, dass Punktwolken neben den Mesh-, den Voxelund den parametrischen Modellen quasi zu einem Formatstandard geworden sind. Semantische Merkmale sind für die automatische Verarbeitung der Punktwolken, z. B. in einem Bauwerksinformationsmodell, notwendig. Die semantische Erweiterung der Punktwolkeninformationen wird aktuell meist händisch durchgeführt und eine Automatisierung (z. B. mittels Deep Learning Verfahren) ist Gegenstand der Forschung. Insbesondere haben sich für diese Aufgabe Künstliche Neuronale Netze (KNN) als effektiv erwiesen, wenn die Daten und Hyperparameter optimiert sind. In dieser Arbeit wurde am Beispiel des KNN PointNet erforscht, welche Punktwolkendaten und Hyperparamter optimal sind. Die Erstellung von Trainingsdaten mit händischen Annotationswerkzeugen, die Implementierung und Erforschung von Prozessen zur automatischen semantischen Segmentierung, sowie die Entwicklung eines heuristischen Qualitätsmodells zur Evaluation von Punktwolkendatensätzen und von semantischen Segmentierungsprozessen standen im Fokus. Das Annotationswerkzeug Point Cloud Classification Tools (PCCT) wurde entwickelt, mit dem die Automatisierung, die Trainingsprozesse von Annotatoren und die Funktionen in Annotationswerkzeugen untersucht werden. Bei der automatischen Punktwolkenverarbeitung sind die Einflüsse Punkte aus fehlerhaften Messungen, Klassenungleichheit und die semantische Klassendefinition zu berücksichtigen. Verschiedene Klassendefinitionen und Methoden für die Minimierung der unterschiedlichen Klassengrößen wurden entwickelt, Adaptionen bei der Punktwolkenvorverarbeitung wurden angewendet und die Gewichtung von seltenen Klassen wurde optimiert. Die Forschungsergebnisse zeigen, dass optimale (datenbasierte) Hyperparameter für die semantische Segmentierung eines Bauwerksdatensatzes definiert werden können. Diese Hyperparamter und das Vorgehen können als Richtlinien für ähnliche Projekte verwendet werden. Eine Steigerung der semantischen Genauigkeit um bis 50% (Recall) ist bei selten vorkommenden Klassen kann durch eine algorithmusbezogene Klassendefinition und die Berücksichtigung der Klassengrößen erzielt werden. Mittels des heuristischen Qualitätsmodells können verfügbare Trainingsdaten und semantische Segmentierungen evaluiert werden. v Table of Contents Danksagung i Abstract iii Zusammenfassung v List of Abbreviations ix List of Figures xii List of Tables xvi 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Research gaps in semantic segmentation of point clouds . . . . . . . . . . . 3 1.3 Research objectives and questions . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 State of the art 7 2.1 Recording systems for point clouds . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Big data and machine learning . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.2 Data pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.3 Clustering and machine learning . . . . . . . . . . . . . . . . . . . . . 13 2.2.4 Deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.5 Evaluation scheme and metrics . . . . . . . . . . . . . . . . . . . . . . 18 2.3 Manual semantic segmentation for point clouds . . . . . . . . . . . . . . . . . 20 2.4 Training data for point cloud applications . . . . . . . . . . . . . . . . . . . . . 24 2.5 Machine learning methods for point clouds . . . . . . . . . . . . . . . . . . . 25 2.6 Deep learning methods for point clouds . . . . . . . . . . . . . . . . . . . . . 28 2.6.1 Semantic segmentation with 2D projection-based deep learning methods 29 2.6.2 Semantic segmentation with 3D grid-based deep learning methods . . 32 2.6.3 Semantic segmentation with 3D point-based deep learning methods . 34 3 Connections of research publications 39 3.1 PAPER 0: PCCT: A point cloud classification tool to create 3D training data to adjust and develop 3D ConvNet . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 PAPER 1: Classification of erroneously measured points in 3D point clouds with ConvNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3 PAPER 2: Evaluating the quality of semantic segmented 3D point clouds . . 40 List of Figures 27 Overview of the connections of the four research publications. PAPER 0: Development of a browser-based classification tool. PAPER 1: Development of a workflow for semantic segmentation with PointNet and investigations on the influence of class Erroneous points. PAPER 2: Development of a quality model for the creation and evaluation of semantic point clouds. This quality model is used for the evaluation of the tool in PAPER 0 and the workflow in PAPER 3. PAPER 3: Extension of the workflow from PAPER 1 and development of methods to optimize the dataset for DL applications. Investigations of the development on the PointNet algorithm. . . . . . . . . . . . . . . . . . . . 43 28 Central issues for improvement in available point cloud annotation tools: Data security, multi-user-capability, segmentation and classification functions, and automation of sub-operation steps. . . . . . . . . . . . . . . . . . . . . . . . . 49 29 Process of semantic segmentation of point clouds serving as an abstract model of the reality. Taken from PAPER 2 and adapted. . . . . . . . . . . . . 54 30 Quality model for semantic enhanced point clouds. Seven relevant characteristics with descriptive quality parameters are shown. Classification of necessary parameters for: Manual segmentations (filled blue circles), manual training data generation (unfilled blue circles) and automatic semantic segmentation (filled green circles). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 31 Converting the quality model into an evaluation matrix for use on datasets, annotation tools, automatic semantic segmentation, and development monitoring. 57 32 Concept for a workflow to apply DL methods for semantic point cloud segmentation. Three modes for the procedure: Training, evaluation and application. . 59 33 DHPs for semantic point clouds. The DHPs can be distinguished according to structural, semantic, geometric and spectral characteristics. A selection of the most common DHPs for each property is summarized. . . . . . . . . . . . 62 34 Semantic segmentation accuracy (IoU) of four common network architectures for the dataset: Semantic3d.net [40]. Selection of four from eight classes of this dataset. The class Scanning Artifacts, which is equal to the class Erroneous point, can be detected poorly compared to the larger classes. Values are taken from the leader board of [40]. . . . . . . . . . . . . . . . . . . . . . 63 35 Comparison of semantic accuracy (recall and precision) on the point cloud of the HafenCity (outdoor) dataset: a) Without the class Erroneous points and b) With the class Erroneous points. Selection of three classes that have different frequencies in the dataset. Data from PAPER 1. . . . . . . . . . . . . . . . . 63 36 Step-wise semantic segmentation for improved differentiation of classes with similar features. With network A, a segmentation is performed for general classes, which is refined in network B. . . . . . . . . . . . . . . . . . . . . . . 64 37 Dataset optimization methods for semantic point cloud segmentation: a) Dataset expansion by randomly copying points, b) weighting the loss function, and c) dataset expansion by copying inputs with infrequent points. . . . 65 xiv List of Figures 38 Process for creating an adjacency matrix and applying it as a network input. . 68 39 Eigenvalue based features calculated from geometric features (x, y, z): a) GT semantic segmentation. b) Sum of eigenvalues as feature. c) Planarity as a feature. d) Linearity as a feature. A histogram is shown next to the legend. . 69 40 Semantic point cloud for the classes Objects and Erroneous points. The semantic segmentation is performed using the PointNet-based workflow with the features: x-, y-,z-coordinates, sum of eigenvalues,planarity and linearity. . . 70 41 Semantic point cloud for the classes Building parts and Interior. The semantic segmentation is performed using the PointNet-based workflow with the features: x-, y-, z-coordinates, sum of eigenvalues,planarity and linearity. . . . . 70 42 Graphical Abstract: Evaluation of semantic segmentation methods using the quality model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XLI 43 Graphical Abstract: The point clouds are separated into different semantic combinations for the training (first row). Different methods are used to extend the class distribution (second line). A DL algorithm is used to train the combinations and extensions, which are then evaluated according to fixed evaluation criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LXXXIII xv List of Tables 1 Commercial tools for semantic segmentation of 3D point clouds, which are not related to a specific scientific work. Abbreviations: Bounding box (BB), offline tool (OT), web service (WS). . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2 Open-source software for semantic segmentation of 3D point clouds, which are not related to a specific scientific work. Abbreviations: Bounding box (BB), offline tool (OT), web service (WS) and Robot Operating System (ROS). . . . 21 3 Parameters of the hardware and software used for development and testing (single workstation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4 General HPs for CNN architectures. Optimized set of HPs and typical values ranges for these HPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5PointNet-specific HPs. Optimized set of HPs and typical values range for these HPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6 Contribution to Paper No. 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . XXVII 7 Contribution to Paper No. 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . XLI 8 Contribution to Paper No. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . LXXXIII 9 Contribution to Paper No. 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . CV xvi 1 Introduction The digitization of everyday life is a trend that has accelerated in recent years, particularly as a result of the global Corona pandemic. New ideas on how everyday life and the working life can be digitally designed have been developed and brought to market maturity in a very short time [1]. These applications frequently use data that represents the real world, as an abstract and geometric copy. Creating a geometric model of real-world objects (e.g., building components, structures, countries, continents) is a core competency of surveyors, and has been the basis for maps, three-dimensional (3D) visualizations (e.g., globes), and knowledge [2, 3]. With very fast, precise and easy to use measurement systems for surface recordings, digitization can be preformed much faster, but usually only the geometry and not the semantics is recorded. In this thesis methods are investigated, which allow to generate semantic information from geometric (and sometimes from spectral) measured values. The basis are point clouds, which are recorded with Light Imaging, Detection and Ranging (LIDAR) scanners and depth imaging cameras. The point clouds are semantically enhanced by machine learning (ML) and deep learning (DL) methods. In particular, the constructed environment, i.e., buildings, cities, and long-stretched infrastructure structures, are objects for which semantic segmentation is necessary [4, 5]. The motivation for a reliable semantic segmentation is explained in section 1.1. The two following sections 1.2 and 1.3 explain the Research Gaps (RGPs) as well as the research objectives (ROs) and, the research questions (RQs). In the last section of the introduction (section 1.4), the structure of the thesis, the relationships between the sections and the form of presentation are described. 1.1 Motivation Cadastre, Geographic Information Systems (GISs) and Building Information Models (BIMs) are the most used applications to represent the real world in an abstract (digital) model and to use them for answering specific issues form topic such as land use, building condition, or mass determinations. These data collections are the basis for public action of administration and economy, strategic planning of social developments and political decisions, so that they are of importance [2, 6]. In the data collections semantic, topological, thematic, spectral and object-inherent characteristics are combined with geometric and geographic object characteristics. Traditionally, these data collections are organized in two-dimensional (2D) representations (e.g., maps or images) in combination with registers (e.g., property registers, land charge registers or land registers). With the advantage of GIS and digital user tools, a paradigm shift has occurred towards the direct storage of object-related information in the form of attributes of a model [6]. A BIM is a data model that represents, among other characteristics, in particular the geometric characteristics of objects as a volumetric 3D model. The great advantage of a BIM 1 1. Introduction is that the details of geometric characteristics and object information can be represented in a scalable and hierarchical manner. In a BIM, which originally comes from planning, the dimensions of the building objects become more detailed and semantic information become more accurate as the planning proceeds. At the beginning of the planning it is only known that a room needs a door and approximately on which wall it has to be, the position of the door, its shape and materials become more concrete as the planning advances. This is currently represented by the five Level of Development (LoDev), [6, 7]. These LoDev, can be further refined for respective characteristics, such as Level of accuracy (LoA), information content or Degree of Modeling [8, 9]. BIM properties can also be used after completing the construction of a building, such as for comparing the as-built planning with the as-is execution (final survey) [10]. In the operation of a construction, BIM is a data format that can be integrated in construction maintenance programs, contributing to the effective and efficient use of a building or infrastructure [11, 12]. Applications include indoor navigation and improved space utilization [13, 14], as well as building control, repair-planning [15], and emergency exit simulation [16, 17]. If data for a BIM is not available from the planning or does not match the as-is status, the data is usually surveyed by total stations, photogrammetric or LIDAR systems [6, 18]. Since most methods scan a surface, this process is commonly called Scan2BIM.Scan2BIM or more generally Scan2Model describes the procedure from the recording to a complete model of the real world in an accuracy and level of detail arising from the application [19]. In the Scan2Model process, the point clouds are usually combined with other recordings and, if necessary, the calculation of the point cloud is carried out for photogrammetric systems. In terrestrial laser scanning (TLS), which is currently the standard method for most highaccuracy models, the registration is done with common points in the overlapping areas of the scans [18]. Mobile Multi Sensor Systems (MSSs), such as scanner backpacks [20, 21] or vehicle-based systems [22, 23] usually use trajectory to connect individual scans, but may also be supported by common points. In most application, filtering is used to partially remove the mismeasurements. The next step is semantic segmentation. Semantic segmentation can be combined with modeling, if a direct automatic or manual creation of parametric geometries is done [24, 25, 26]. These methods are used in applications to create floor plans [27] or surface models (e.g., meshes or voxels) [28] from the point clouds. These models are usually not transferred back into the data format point cloud, but they form parameterizable geometries, such as cylinders, cubes, planes, lines or circles, which are used to build complex object-oriented models, such as GML [29] CityGML [30], IndoorGML [31] or Industry Foundation Classes (IFC) [32]. The methods are usually very specific to an application and require detailed prior knowledge about the data and the task. The most common applications for these methods are the modeling of building structures. However, in most applications, semantic segmentation and modeling are performed independently. The semantic segmentation of point clouds is the most complicated step of this process chain to automate, as the objects vary in geometric size, shape and the recorded scenes differ significantly [33]. Parameters and thresholds for separation by semantic ob2 1. Introduction jects can be insufficiently defined, which still makes ML and DL most suited methods [34]. The performance of ML and DL varies, depending on the data and the complexity of the class definition according to which the point cloud should be segmented [35, 33]. However, strictly point clouds are imperfect data for semantic segmentations, since the training data are usually only available to a small extent, do not have a homogeneous structure, and are erroneous [35]. These disadvantages of point clouds lead to varying semantic accuracies for different classes, to systematic confusions between classes and to unfavorable foundations for modeling [36]. Overcoming the imperfection and understanding its causes for the case of semantic segmentation with DL for building reconstruction is the motivation of this thesis. The aspect of processing point clouds with DL, the quality of point clouds and the generation of training data from point clouds must be examined in a structured manner as it is explained in [37, 38, 39, 40, 41, 42, 43]. Like [36], this work focuses on the point clouds and its weaknesses, as well as methods to overcome them. 1.2 Research gaps in semantic segmentation of point clouds DL is the most suitable method for the semantic segmentation of point clouds but it has several downsides and aspects that are less researched. The main researches on DL methods deal with the following aspects: • Optimization of algorithms and network architectures [44, 45]. • Enhancement of the benchmark datasets collections [40, 46, 47, 48]. • Neighborhood representation for algorithms input [49, 50, 51]. • Automatic transformation of point cloud information into parameter models [35, 52]. • Optimization of manual annotations [53, 54, 55]. • Investigation of the impact of point clouds and its pre-processing for optimal semantic segmentation [36, 56]. • Development of quality models and characteristics for semantic point clouds [18]. • Concatenation of DL with ML [57, 58]. The findings for one research aspect sometimes provide the foundation for the others. This can be seen in the example of the development of the network architecture of PointNet [45]. This network architecture enables an efficient and direct processing of larger point cloud scenes (>1 million points). Now, the data pre-processing of the point cloud format is no longer a primary issue, but the data content is a new issue. The main research in the field of DL methods on point clouds does not address real and practical applications, therefore many relevant and influencing parameters are neglected. This leads to the RGPs addressed in this thesis: 3 1. Introduction RGP 1: Influence of dataset characteristics, points and point clouds for semantic segmentation. RGP 2: Development of a heuristic description and evaluation of semantically segmented point clouds. RGP 3: The development of a workflow for semantic segmentation of point clouds in building modeling processes. The research objectives (ROs) are derived from the RGPs, but do not necessarily address the entire research gap. How the RGPs can be closed is explained in more detail in section 1.3 based on the ROs and the RQs. 1.3 Research objectives and questions The identified RGPs are in the overlapping field of the disciplines of computer science, mathematics, data science, computer version, civil engineering, facility management, as well as geodesy and geoinformatics. In order to close these gaps, innovative data models and processing algorithms must be implemented by means of modern high-performance computer systems to address topics arising in the digitization of buildings. Recorded digital datasets of buildings have measurement errors, vary in terms of semantic class sizes, include fine and coarse objects in unstructured and heterogeneous point clouds (Figure 1). Figure 1: Measured TLS point cloud with segmentation and annotation errors. Class Erroneous points in red and class Object in blue. These datasets are not optimal for processing with ML or DL methods, due to the data content and data format. Nevertheless, ML and DL methods are the most efficient and accurate methods for semantic segmentation if the data is homogeneous, structured, and arranged in a raster. In order to harmonize characteristics of point cloud datasets and DL algorithms, the following three ROs are tackled: 4 1. Introduction RO 1: Evaluation of methods for the manual annotation of point clouds regarding efficiency, usability, accuracy, and the development of an experimental annotation tool. RO 2: Development of a quality model that heuristically describes semantic point clouds. RO 3: Development of a workflow for semantic segmentation in order to investigate the influence of point clouds content and format in DL methods. RO 1 can be achieved by explaining the developing steps of the Point Cloud Classification Tool (PCCT) (PAPER 0) and the investigations of its usability. For manually annotated point clouds with a computer, the users must have segmentation tools, a visualization of the point cloud (on a screen), guidelines for the classification process, and tools for the classification. Based on these statements, the following RQs should be answered: RQ 1.1 Which annotation tools (manual segmentation) for point clouds exist? What functions can be found in these tools? How efficient, reliable and effective are these tools and how can these characteristics be determined? RQ 1.2 Which annotation tools can be used for the semantic segmentation of challenging real-world indoor TLS point clouds? RQ 1.3 How can semantic segmentation tools for point clouds be enhanced and improved? RQ 1.4 How to become a good annotator for semantic point clouds? How can the performance of annotators be measured? What do annotators need and how can the tool support them? In order to answer the questions of RO 1, a heuristic quality model must be used. The development of a quality model is the RO 2. The quality model evaluates the semantic segmentation process and the semantic point cloud. The development of the model is guided by following RQs: RQ 2.1 What are suitable semantic point clouds? What are the characteristics of point clouds? How can the characteristics of the point cloud be determined, measured and compared? RQ 2.2 How is a quality model for semantic point clouds designed? Which parameters are necessary for the description of the characteristics? Does the quality parameters differ for annotation and automatic semantic segmentation? RQ 2.3 How can the quality model be applied for the semantic segmentations of building point clouds RO 3 is based on RO 1 and RO 2 and is the realization with the workflow for semantic segmentations. The training data created by the PCCT or other tools and the performance evaluation of the workflow by the quality model are necessary to answer the RQ 3.1 to RQ 3.3. The workflow is developed for the point clouds created by TLS with imperfections as 5 1. Introduction shown in Figure 1. In the workflow, established DL methods are integrated. The formal and content influencing parameters of point clouds are evaluated in experiments. The RQs which guide the development are: RQ 3.1 How can DL methods be integrated in a workflow for semantic segmentation of point clouds? Which DL methods are suitable? RQ 3.2 Which hyperparamters need to be defined for applying PointNet in a semantic segmentation workflow? How are the values for these hyperparamters determined? RQ 3.3 How can the influence of the dataset be controlled by data-based hyperparameters in the semantic segmentation of point clouds? What are the main influences? The three central ROs are covered by three peer-reviewed and one extend-abstract-peerreviewed publications. There is no one-to-one assignment of one RO to one publication. Instead, single or multiple RQs are covered in each publication. The connections between the publications and the ROs are explained in section 3.5 and Figure 27. 1.4 Outline of the thesis This cumulative dissertation consists of a framework thesis (sections 1 to 5) and the four publications in the appendices A (peer-reviewed publications) and B (non-peer-reviewed publication). The framework thesis presents the state of the art, the terminologies (section 2), the connections between the individual publications (Section 3.5), and the ROs, RQs, and results (section 4). Section 5 summarizes the key conclusions and outlines further research approaches. PAPER 0 is in section B.1 and PAPER 1 to PAPER 3 are in sections A.1 to A.3. A reference to the publications is made by the indication PAPER #. References are used to avoid repetitions of results, proofs, and detailed descriptions that have already been discussed in the publications. For better comprehension, conclusions and general observations are discussed in section 3 and in the individual RQs (section 4). 6 2 State of the art Semantic point clouds are the foundation for modeling complex environments. Their deployment covers the entire process with the acquisition, the parameter-based filtering and the semantic segmentation of the point cloud (steps 1 to 3 in Figure 2). Based on the semantic point clouds, parametric, solid models, Computer Aided Design (CAD) and BIM models are created. These models are used in GIS [59], construction management applications [12, 60], and in private and public registers [61] (steps 4 and 5 in Figure 2). Figure 2: Process of creating semantic point clouds. Recording of point clouds with an optical recording system. Registration of the individual recordings, resulting in a complete point cloud. Semantic segmentation according to given classes set. Modeling of objects in the semantic point cloud. Implementation of the models into an application. Taken from [62] and adapted. In context of buildings, LIDAR scanners and measurement cameras are used to record entire surfaces in a fast way. The working principle, the differences of the measurement system as well as its influence on the point clouds are explained in section 2.1. The recording and the semantic segmentation are significant for the quality of the semantic point cloud. Semantic segmentations are performed manually as well as automatically and due to the size and complexity of the data this topic comes under big data.big data applications require special data handling, which is carried out with ML methods. The basics of big data and ML are introduced in section 2.2. The process of manual semantic segmentation is in section 2.3. ML models learn the relationship between input and target data from the data itself. The characteristics of training data are described in section 2.4. The state of the art of automatic semantic segmentation of point clouds is described by ML and DL methods in sections 2.5 and 2.6. 2.1 Recording systems for point clouds Point clouds have become a quasi-standard for storing recordings and visualizing the surfaces of real objects in the digital domain. This quasi-standard can be explained by the fact that many recording systems have to store the measured values very fast (data-stream) and which does not allow an order (sorting) according to the contained object classes [63]. 7 2. State of the art based methods. In addition, a distinction is made between knowledge-based (black) supervised (orange) and unsupervised (green) methods. Figure 8: Summary clustering and semantic similarity segmentation methods. Knowledgebased clustering methods (black), data-based clustering methods (green), and data-based semantic segmentation methods (orange). In programming typically, knowledge is used to process and analyze data (knowledgebased). This requires that the data content is known and can be selected via parameters such as thresholds or number of objects searched. If conditions are met, clusters can be formed using threshold selection [95] (e.g. threshold > a spectral value) or fitting a geometry in the point cloud. Methods such as Random Sample Consensus (RANSAC) [96, 97], require some parameters, such as number of objects and its shape, and randomly search the data for these patterns to form clusters. In addition to parameter-based methods, which are highly dependent on a-priori knowledge, data-based methods are an alternative that learn the relationship between data and target class from the data itself. Three categories for data-driven approaches are described in the literature. These are supervised learning,unsupervised learning, and reinforcement learning. In supervised learning, the target variables are known and the algorithm learns or determines the relationship between features and the target variables. In unsupervised learning, no target variables are given and a fix or an unspecified number of clusters with high similarity in the features is formed. Reinforcement learning is based on the idea of trial-and-error. The algorithm performs the classification task many times and gets feedback at the end of each pass indicating whether the classification is correct or incorrect [98].The last method is usually not used for semantic segmentation. 14 2. State of the art Unsupervised learning is used primarily for clustering data objects. Thereby, differences and similarities in the features are determined e.g. via static method, feature orders or transformation in another feature space [99]. Discriminating methods, such as edge detection [100] or Principal Component Analysis (PCA) [101, 102], define differences by boundaries in the feature space. Based on these boundaries (or thresholds), the unlabeled clusters are formed. Generative unsupervised methods, such as graph-based methods [103, 104], Regional Growing (RG) [105], or k-Means [106], start at one or more starting points and grow around the object points that have the greatest similarity. The resulting areas and structures are the unnamed clusters. Using user knowledge or data-based methods, the unnamed clusters become classes. Supervised learning methods, such as Naive Bayes [89, 107, 108], Logistic Regression [89, 109], k-Nearest-Neighbors (kNN) [86, 110, 111], Support-Vector-Machines (SVM) [86, 112, 113], and Decision Trees (DTs) [108, 114] use target variables to optimize the learnable model parameters. In addition to the learnable parameters, each ML method has additional parameters that must be specified prior to training. These are called hyperparameters (HPs) and include the algorithm itself, the proportion of training and test data, learning rates (LRs), and stopping criteria. The HPs are discussed in PAPER 3. The previously mentioned methods are mostly classified as weak ML. They allow direct semantic segmentation for a predefined defined set of classes. The labeled data is needed for this purpose (section 2.3). The adjective weak refer to the fact that the features are used directly for semantic segmentation and no depth features are formed from the raw features. This requires that the necessary independent variables have been optimally chosen and that the features are free of gross errors. Data pre-processing has an even larger impact than in DL [86]. To make the methods more robust for varying data, Ensemble Learning (EL) methods such as Random Forest (RF) [115] were developed. RF use multiple DTs and all are trained under different conditions. The results are combined using methods such as voting, bagging, stacking or boosting. EL also uses different combinations of independent variables to minimize the influence of correlated or irrelevant independent variables. The EL leads to a measurable increase accuracy for most applications [86, 116]. 2.2.4 Deep learning DL methods are usually more robust to errors and major changes in features than ML methods, such as SVM or RF. They have become very important with the rise of big data, as they find hidden patterns in large and complex datasets [117]. Commonly, DL is used as a synonym for Artificial Neural Networks (ANNs). The functionalities, the different types, as well as the advantages and disadvantages of ANN are briefly explained in this section. The use of ANN for semantic segmentation of point clouds will be discussed in more detail in section 2.6. An ANN is a mathematical-technical model of a natural neural network such as those found in brains [118]. There are static and dynamic components in the ANN, which are controlled 15 2. State of the art by the initial HPs. The static components are the processing unit (neuron), the connections (weights) and the network topology (network architecture). The learning phase and the processing phase are the dynamic components [117]. Figure 9: Neuron and network architecture: a) Neuron architecture and function. a) Simple ANN with input and output layers. b) ANN with a hidden layer. Inspired by [119, 120] The neuron is an independent unit that performs a partial operation of the network. The neuron processes the numerical information by aggregating its input and calculating a new actuation value with a (typically nonlinear) function (Figure 9a). Step-,Sigmoidor ReLufunctions are used. Neurons are organized in layers and forward parts of the information to other neurons. The simplest ANN consists of only two layers and can only be used for linear problems (Figure 9b) [120, 121]. The input layer has as many neurons as there are independent variables in the dataset and forwards them to the output layer or, in more complex networks (as in Figure 9c), to the hidden layer. In the output layer, there is one neuron for each target variable. The layers are connected by weighted and directed graphs. If all neurons of one layer are connected to all neurons of the next layer, this is called fully connected (FC) layer. Also, sparsely connected (SC) layers where some neurons have connections are frequently used. Information can flow in all directions. In practice, for static classification tasks, the feed-forward (FF) architectures have become most popular. The FF architectures feed information from the input layer through all hidden layers to the output layer. The output layer provides a quasi-probability for each class. A classification function (e.g., Softmax) is used to perform the interpretation of the output layer results. During the learning phase, a large number of features along with the labeled class (training data) are fed into the ANN and after each pass, the loss is determined across all learning samples. By comparing network predictions and target data, the network loss is determined. This loss needs to be minimized by optimizing the weights on the graphs using back-propagation [119, 120]. After the network has been trained several times and the loss value is minimized, the ANN can be tested with independent data (section 2.2.5). Once the test parameters are finalized, the ANN can be used in the processing or inference phase [98, 117, 122, 123]. A special type of ANN uses equal weights for all inputs (share weights). These inputs are images or 3D data and carry information in the arrangement of features (neighborhood dependent data). Commonly, ANNs for this kind of data are called Convolutional Neural Net16 2. State of the art works (CNNs) [124]. Using CNN, each feature of each variable is loaded as a 1D, 2D or 3D tensor. For each network feed, there are as many tensors as there are variables in the first layer. To extract depth features, each tensor is multiplied by weights of a feature map (F-map) and these products are summed up, so that a new depth feature is created from all input variables. The F-map is a tensor, with a fixed width and length (usually a few entries large), that is shifted over the input tensor such that the new features remain local. There are multiple F-maps for each convolutional layer (Conv layer), so several feature variables are given to the next layer. The F-maps correspond to the weights in the ANN. When the F-map is longer and wider than one entry, the width and length of the net feature tensor will be reduced (Figure 10). Stronger features are formed by convolution and pooling layers. Commonly, the classification step is done with a FC layer [71, 118, 120, 125]. Figure 10: Function of the one Conv layer at a CNN. Inspired by [120]. Figure 11: Common CNN-Architectures for semantic segmentation. a) Encoder-Network and b) Encoder-Decoder-Network Special CNN architectures are the shared Multi Layer Perceptron (MLP), Encoder-Networks (EN), and Encoder-Decoder-Network (EDN). These are often used in semantic segmentations. The MLP is strictly an ANN, such as in Figure 9c, and unit to extract features from data inputs. By implementing this unit with a 1D CNN, more operations with identical weights can be performed in parallel [126]. EN encode the input data to depth features by chained 17 2. State of the art Conv layer and used at the end a FC layer to classify each point (Figure 11a). This method is used for sparse point clouds or classification questions [127]. In the EDN, the features are encoded and summarized in the encoder phase. In the decoder phase, the features are expanded to the number of input points and decoded (hierarchical approach) [128]. Features can be shared between encoder and decoder layers of the same size through connections (Figure 11b). An alternative way to distribute information between inputs is to use Recurrent Neural Networks (RNNs). RNNs inherit information from previous inputs to the current input and subsequent inputs. The value of the previous information become lower over the time (Figure 12). RNNs are mostly implemented in the form of Long Term Short Memory (LTSM) networks, which are explained in [129]. Figure 12: ANN with a recurrent layer. The outputs of the recurrent layer is used as additional input in the next pass. With time the inputs become less meaningful, so that its influence is lowered via weights. Compared to most other ML methods, ANN and CNN have a high learning capacity, when large training datasets are available. They can be efficiently adapted to new tasks, once the infrastructure for training is set. They usually generalize better than ML methods and are more robust of errors. Disadvantages of DL are long training times, lack of to small traceability of the learnable parameters and there is a need for large amounts of training data [117, 120]. 2.2.5 Evaluation scheme and metrics Most algorithms use intermediate classification results to optimize the learnable parameters, thereby validation is already part of the learning. This validation is done using only very few metrics, which mostly describe the semantic accuracy. Typically, in supervised learning, Overall Accuracy (OA) and loss are used. In reinforcement learning, binary answers (false or true) are given. In non-supervised learning, no validation occurs during learning in this 18 2. State of the art sense [89]. The validation during learning gives insufficient information to evaluate the performance of the trained model on new similar data and for each individual semantic class. Before a model can be productively applied, a full evaluation of the model with unknown data must be performed. This must provide information on semantic sensitivity (recall) and specificity (precision), evaluate the choice of HPs, and provide other metrics such as geometric accuracy [108]. The basis for the evaluation is a ground truth (GT) dataset that is used to validate whether the classification for each data point is correct. If this is the case, the point is considered to be true positive (TP), if not, the point is considered as false positive (FP) in the predicted class and as false negative (FN) in the true class. This classification of points is usually presented in a confusion matrix [90] (Figure 13) and is the basis for computing other semantic metrics, which [108] describes in general terms. A review of metrics in point clouds is done in PAPER 2. Figure 13: Confusion matrix for the example of three classes. TP = true positive, FP = false positive and FN = false negative. TP of the one classes is equal to true negative (TN) for all other classes. The automatic classification methods have a large number of HP that have to be customized. The correct choice of HP is the prerequisite for optimizing the learnable parameters and succeeding in classification. In a broader sense, the training of the model is not complete after training of the learnable parameters. Rather, this is only one pass of the integrative optimization of the HPs. This optimization with various manual and automatic methods is presented for general models in [90], for ML methods in [86, 108], and for DL methods in [130]. In PAPER 3, DL methods are reviewed in detail. Evaluation using non-semantic metrics is necessary for special (e.g., geodetic) issues, but is rarely presented in the literature. 19 2. State of the art 2.3 Manual semantic segmentation for point clouds Semantic point clouds are the basis for creating surface models [4], developing BIM applications [131], building the navigation basis for autonomous vehicles [47], and developing algorithms for automatic semantic analysis of 3D point clouds [39, 132]. Manually enhancing point clouds by segmenting the individual objects in the point cloud and assigning a label is named as point cloud annotation. Point cloud annotation is a very complex task that is time consuming and most often performed by experts [133]. To speed up this task and allow less experienced annotators (e.g., crowd workers) to do this, various software tools have been developed to make annotations more reliable and simple. A brief summery of these tools and providers of these services (Data as Service) is given in the following. These tools and their described functionalities form the basis for the PCCT. The motivation of the PCCT is to produce independent, reliable, fast and without additional costs test data for examinations, as there were only few similar tools available at the beginning of this thesis (section 3.1). The literature review on various manual (open-source and commercial) tools for semantic segmentation of 3D point clouds shows that eight properties of the tools are relevant. These properties are visualization of the point cloud, big-data-capability, tools for segmentation, multi-user capability, adaptability to new circumstances, feedback capability for annotators, semi-automation, and annotation evaluation. An overview is given in Figure 14. A selection of the reviewed tools for semantic segmentation of point clouds, showing methods diversity, is presented in Tables 1 and 2. Figure 14: Requirements for a point classification tool. 20 2. State of the art Table 1: Commercial tools for semantic segmentation of 3D point clouds, which are not related to a specific scientific work. Abbreviations: Bounding box (BB), offline tool (OT), web service (WS). Tool name Selection Application OT / WS (AutoCAD) Recap [134] Freehand, filter, fit, polygon TLS OT PointCab [135] Freehand, filter TLS OT AWS SageMaker [136] By own design All WS basic.ia [137] BB, semi-automatic Auton. driving WS, scale [138] BB, semi-automatic Auton. driving WS Point Cloud Technology [139] Data as Service All WS Table 2: Open-source software for semantic segmentation of 3D point clouds, which are not related to a specific scientific work. Abbreviations: Bounding box (BB), offline tool (OT), web service (WS) and Robot Operating System (ROS). Tool name Selection Application OT / WS Cloud Compare [140] Freehand, filter, fit, polygon All OT MeshLab [141] Freehand, polygon All appl. OT Multi-Label PC [140] RG All appl. OT / ROS Go Then Tag [142] Solid fit, pencil All appl. OT PC Annotate [55] Solid fit TLS, Auton. driving OT SemanticKITTI [53] Freehand, brush Auton. driving OT 3D Annotation [143] BB, semi-automatic Auton. driving OT LATTE [144] BB, semi-automatic Auton. driving OT 3D BAT [133] BB, semi-automatic Auton. driving WS SAnE[145] BB, semi-automatic Auton. driving OT Visualizing 3D data and navigating through it on a two-dimensional screen is described by [146] as a central problem, because the data can only be seen from one perspective, which leads to mistakes in interpretation [147]. [146, 147] address this problem by visualizing the data on a 3D display wall and use a touch screen table for navigation. The idea of processing 3D data in a 3D space is also addressed by the PointAtMe application [148], which uses virtual reality (VR) glasses for visualization. The annotators wear VR-glasses and can move freely through the point cloud. The annotators segment and classify individual objects via the controllers by placing a bounding box (BB) around the points belonging to an object. All other tools from Tables 1 and 2 use a standard 2D screens for the semantic segmentation on which the point cloud is displayed in a predefined perspective [143] or as a free navigable model. The free-perspective choice is default. The free perspective choice is advantageous for manual segmentation of objects of different sizes. This option allows to look from any angle and at any zoom level at the areas to be processed. However, using this option requires that the point cloud can be loaded in a very high resolution, ideally without delays. Fast loading is an aspect that concerns big data capability and is usually implemented by splitting the point cloud into 3D tiles. The 3D tiles are usually realized by kd-tree or octree methods. These methods organize the point cloud hierarchically, so that only the necessary data section is completely loaded at any given time. The 21 2. State of the art methods kd-tree [149] and octree [150] are state of the art in mass data processing [142]. The choice of a hierarchical structure has a great advantage for visualization, because the point cloud can be used in full detail. However, for segmentation, this partitioning can be disadvantageous, because during segmentation the storage structure is changed and has to be recalculated again. In practical applications (e.g., Recap [134]), it is observed that these calculations can be reduced if only all points of other classes are deleted from an existing data structure. By deleting the points, the existing data structure remains unchanged and does not need to be recalculated during segmentation. Loading the point clouds with all features into the working memory (direct user access) is very time-consuming, so in many applications only parts of the dataset can be loaded and processed at any given time. The coarse subdivision is usually done according to semantic aspects, such as roads [53], measurement drives [47], recording stations or rooms [27]. Seldom, permanent database systems (e.g., MariaDB and PostgreSQL) are used for benchmarks, because the data is meant to be exchanged. In addition, folder-based data storage, portable databases such as 5h or SQLite are sometimes used. These formats have the advantage that the data can be loaded via Structured Query Language (SQL) commands efficiently by several users at the same time. Besides solutions for temporary and permanent storage of point cloud data, the filtering of the point cloud according to point cloud density or geometrical aspects is an important aspect. Many manufacturers of recording systems offer optimized parameter-based filters in their own software for point cloud pre-processing or general static filters, such as Statistical Outlier Removal (SOR) [151] or Voxel-Subsampling and Fast Cluster Statistical Outlier Removal (FCSOR) [152]. It is important that the geometry of the object is not changed beyond what has been done by the recording system and that, known measurement errors are minimized. The annotation of the point cloud consists of segmentation and classification. Traditionally, for segmentation, a perspective is selected in which the object to be classified can be recognized well. The object is separated from the environment with a polygon or lasso and assigned to a semantic class [147]. Besides the free-form polygons or lasso using the mouse, the selection of points is often done by brush technique (sweeping over an area with the mouse) [53], placing BBs over the object [145] or selecting by parametric 3D solids [55]. The selection of points by parametric 3D scenes is done purely by humans, who interpret the 3D scenes differently, variate in the degree of careful work, and are different good trained for the task. This is concluded by [143] under the factor of human error. To minimize this occurring factor, segmentation is often considered as a control screw for automation. The approaches for the automatic geometric segmentation can be summarized in five main methods and one mixed method (hybrid methods). These basic segmentation methods are revisited in section 2.5 and used in a modification for the dataset point cloud. The basic segmentation methods are according to [147]: • Edge-based segmentation. • Reginal grow. • Model fitting. • Traditional Machine Learning. • Deep Learning. 22 2. State of the art • Hybrid methods. For a detailed description of the main methods and examples,references are made in Table 1 by [147]. The list can be extended by the object tracking respectively instance datasets, such as applied in autonomous driving [53, 55, 145]. Multi-user capability plays a minor role in many scientific manual semantic segmentation tools as the datasets are mostly shared by the researchers as in [41, 53, 55]. The annotators mostly process one assigned sub-dataset locally with a specifically developed tool or according to a process description [41, 153] for a general point cloud processing program, such as Cloud Compare [154]. Especially when special hardware, as in [148], or extra powerful hardware [134] is used, the scalability by the number of workstations is usually no longer efficient and economical. Multi-user capability is mostly implemented in the scientific context by crowd-working-services (CWS), such as Amazon-Web-Services, also known as Amazon Mechanical Turks [136]. For example, this is propagated in [133] and considered during software development. Few applications [155] are identified that use the AWS or similar services. Commercial service providers, such as basic.ia or scale, provide multi-user web applications for various data classification tasks or deliver ready-labeled data. [55] explain in their discussion of the category of annotation tools, that few information is known about the process, the data accuracy and the data privacy. The commercial annotation services for point clouds focus on the market of autonomous driving. This is done by the BB selection of the data, the initial class sets that primarily include traffic participants, surface types, and street furniture, and the trajectory-optimized visualization and processing. Also, many scientific works, such as [53, 55, 143, 144, 145], are optimized for autonomous driving. However, most of these annotation tools are transferable to mobile mapping applications, because the class selection in these is adaptable or already includes most classes for outdoor applications. Traditionally, indoor point clouds are captured with RGB-D cameras, so semantic segmentation is done with 2D annotation tools, such as LabelMe [156] or the tools described in the review by [157]. Point clouds that are sourced by TLS are predominantly annotated using Cloud Compare or commercial applications, such as Recap [134] and PointCap [135]. These tools are optimized for viewpoint-based recording. Each annotation method is usually developed for a specific dataset (data format) and a specific task, and usually requires major effort to adapt to a slightly different application. The evaluation of manual semantic segmentations and the related feedback and training of annotators are reviewed in detail in PAPER 2. In addition to the statements there, the experiences of [54] can be followed for the training of the annotators. They emphasize the selection of the annotators, the previous experience, an intensive training phase before the proper task and an annotator-bias (individual errors). 23 2. State of the art Figure 20: BEV projection. The point cloud is oriented along the z-axis and transformed into a raster plan with fix a fix raster structure. with a FC network. In the interference phase, the passable area of each scan is semantically segmented in a few microseconds with approximately 90% recall and 90% precision5 [202]. Following this task and approach, [203] have developed a similar method using the LoDNN Network [202] for semantic segmentation. There, in addition to the geometric features, features such as pixel densities are used in order to take generalization into account. Furthermore, WV point clouds are semantically segmented with the U-Net [128], after they have been transformed into a panoramic image. Different resolutions of the panoramic images are evaluated with the U-Net and then the sub semantic segmentations are combined to a joint one using thresholds. Its performance is validated on the SemanticKITTI benchmarks [53, 204] and is close to 90% for recall and precision.SurfConv [205] and PIXOR [206] aim to detect individual objects, such as other cars, pedestrians or cyclists, which are of a particular interest in the point cloud. This is done in the first steps as described above. The semantic segmentation is performed using a CNN chained by FC layer, where the FC layer is used to express the location, orientation, and reliability of a BB that envelops the object. The accuracy of semantic detection varies between an average precision of 55% and 75% [206]. A complete semantic segmentation of WV scenes are intended with the methods SqueezeSeg [58], RangeNet++ [207], LU-Net [208] and SalsaNext [209]. These methods do not differ fundamentally in the scheme of data processing. In all works, the point clouds are projected onto a sphere, which is then unrolled as a panorama (Figure 21a). Furthermore, different improvements for the geometric resolution are developed and applied. The back projection from the image to the point cloud is addressed and optimized by a kNN step [207] and Conditional Random Fields (CRF) [58]. Besides U-Net,Darknet53 [199], SquezzeNet [210], and ResNet-18 [211] are used and adopted. The performance of these methods varies from 52% to 60% 6Intersection over Union (IoU). TLS point clouds and point clouds generated with mobile MSS have a much higher density and cannot be mapped from a single perspective. Points would be missed by occlusions 5Published by the developers. 6Validated on the SemanticKITTI dataset by [46]. 30 2. State of the art Figure 21: Projection of a 3D point cloud into (2D) image. a) Spherical or cylindrical projection. b) Multi-view-image projection and transformation. during semantic segmentation or the geometric context would not be identified. Multi-viewimage approaches (Figure 21b) are developed by [212, 213]. The point cloud is considered as a surface and a mesh is computed from it. [212] use randomly generated images that represent completely the mesh at different distances and rotations. These images are semantically segmented with EDNs, such as U-Net and SegeezeNet, and the semantics are projected back onto the mesh. From this, the semantic information is transferred to the point cloud. The approach of [213] use planes that tangentially intersect the point cloud in one point. Starting from this point, all neighboring points in small area round that the tangential point are projected into the plane, and the areas without information are completed by interpolation between points (Figure 22). The plane is overlaid with a pixel grid and all images are semantically segmented by U-Net. The IoU for these methods varies between 51% and 67%7. Figure 22: Projection of the points onto a tangent plane. Creation of a multi-view image (Simplified 2D illustration). A recent work use 2D CNN for semantic segmentation of 3D point clouds by using the recording profiles [214], fuses the point clouds with other data such as images [215], or uses 3D CNN in air-born laser scanning (ALS) analysis [216]. [214] use a profile laser scanner that 7Validated by developers on the Semantic3D.net dataset [40]. 31 2. State of the art generates a 2D point cloud. This point cloud is transformed into a raster image and evaluated with a 2D CNN. In the data fusion method, [215] use images for the semantic segmentation. A mobile mapping system captures the images synchronously to the laser scans. The semantic information is generated in the images and transformed to the point cloud. This method requires a very accurate synchronization and calibration of the scanner and the cameras. 3D CNN have a wider geometric dimension and convolute the dataset in three directions. This is computationally intensive, so [216] additionally transform the ALS point cloud into orthophotos. In addition to raster-based 2D CNN, lattice representations of the point clouds are used. The lattice approaches use a bilateral Conv layer [217] to transform the point clouds into the 2D structure. They could be processed with 2D CNN. Advantage of these methods is that 3D points and georeferenced images can be fused, as in SPATNet [218], and be used for semantic segmentation. 2.6.2 Semantic segmentation with 3D grid-based deep learning methods In the context of 3D grid-based DL methods, point clouds are converted into 3D raster structures for evaluation with CNNs. These can be voxel-based, tree-based or lattice-based. This intermediate format allows to apply Conv layers convolut in three dimensions (3D CNN). Most of the 3D CNN architectures are based on the 2D CNN architectures [219] applied to raster images. For semantic segmentation, EDN and EN with a FC layer are used, which assign a class to each voxel. By interpolation in the 3D space, the labels are transferred to the points or further refinements of the semantic segmentation are performed. A general pipeline for these methods is shown in Figure 23. The advantage of using 3D CNN is that the information is preserved in all dimensions and the segments are not mixed in the reduced dimension. A majority of 3D CNN networks commonly merge multiple points into one voxel, so that the data is generalized as well (Figure 24a). The main disadvantage of 3D CNN compared to 2D CNN is that the application and training times are significantly increased, since the number of operations is potentiated. This has led to many early architectures consisting of few layers [124] and the voxel structure being transformed into an occupancy grid (Figure 24b) [220]. The EN with FC layer performs a classification for each network input. In VoxNet, the detection of objects in the point cloud is performed with such an architecture [124]. [219] proceed identically, but only label the voxel representation. For the transfer from voxel to point, an intermediate step is introduced that takes into account the distance to the voxel center point. [40, 221] use a sub-voxel grid computed for each point as network input. Small-dimensional Conv layers with 16 x 16 x 16 voxels are used as a basis. In the architecture of [40], the local neighborhood is additionally considered by using voxel grids with five different voxel edge lengths (2.5 cm to 40.0cm). For each of these five voxel grids, a VGG16-like [222] network architecture that is extended as 3D CNN is used. All the sub-CNN results are combined and the classification is performed in the FC layer. 32 2. State of the art Figure 23: Workflow semantic segmentation utilizing 3D grid structures. Top: The entire point cloud is transferred to one grid. Iteratively, several voxels are fed into a 3D CNN. The voxel grid is semantically segmented. The information is passed by interpolations to the point cloud. Bottom: A sub-voxel grid is created for each point. Each sub-voxel grid is classified by the 3D CNN. Each point is directly assigned to one class. Figure 24: Voxel structures (in 2D perspective). a) Regular voxel grid. b) Occupancy grid. c) Octree with refinement due to point cloud density. Other 3D CNN are based on a EDN, where a part of a point cloud is transformed into a voxel structure. The FC network can act as an encoder to detect a specific object in the point cloud. Vote3D [223] and Vote3Deep [224] use such a CNN architecture, embedded with a voting algorithm, to detect the BB of objects relevant to autonomous driving. EDN for spatial classification of RGB-D images find applications in the architectures: SSCNet [225], ScanNet [226], and ScanComplete [227]. The voxel-resolved RGB-D image is given as an occupancy grid with a resolution of several centimeters as input to the CNN. In this case, the key feature is occupancy or non-occupancy state of the voxels (Figure 24b). Through the EDN, the occupancy grid is directly classified and a transmission of the semantic labels is made to a point cloud or a mesh. [227] use a hierarchical-chained architecture to efficiently increase segmentation resolutions. Additionally, this type of architecture can fill gaps created by recording perspectives [225, 227]. The SEGCloud architecture has a CRF layer after the 3D CNN layer, which is used for finer (sub-voxel) segmentation [228]. Thus, combining traditional ML and DL methods contributes to an efficient and easily increase in semantic segmentation accuracy. 33 2. State of the art OctNet [229] uses an octree as voxel structure (Figure 24c). With the octree, the sizes of the voxel cells are adjusted based on the occupancy of the voxels [230]. OctNet is used for semantic segmentation of point clouds representing larger facades. In this context, the height of the facade specifies the maximum size of the octree. The features of the points falling in a voxel are combined by computing an average and calling this value voxel feature. The GT label is determined using the dominant class of the points in this voxel. Semantic segmentation is performed using a 3D EDN, so that directly the selected portion can be semantically segmented. Besides octrees, kd-trees [231] are also used for small point clouds applying FC layer or 1D-convolution for feature extraction and semantic segmentation [232]. 2.6.3 Semantic segmentation with 3D point-based deep learning methods The DL methods from sections 2.6.1 and 2.6.2 have the disadvantage that points always have to be converted into a raster geometry and information is generalized. Direct pointbased semantic segmentation became popular with the development of PointNet [45]. PointNet uses MLPs to extract depth features, which extracted individual for each point. The individual point features are combined via a max-pooling function, resulting in global features describing the dominant features of all points currently fed into the network. A detailed description of of PointNet can be found in PAPER 1 and PAPER 3. Due to the point-wise extraction, the order of the points is not important. However, this also has the disadvantage that neighborly relations, which are described by several points, are not considered in semantic segmentation. Moreover, the global features refer only to the current input, consequently in most cases the features describe only a very small part of the point clouds. In principle, semantic segmentation can be performed using point-based features, thus researchers use PointNet or parts of PointNet frequently. Besides PointNet-based enhancements, which will be discussed in more detail below, RandLANet [44] is one more recent developed network for semantic segmentation of large point clouds. In RandLANet, the point clouds are semantically segmented in one step by randomly reducing them. The key weaknesses of PointNet concerning neighborhoods are directly addressed in [233, 234, 235, 236, 237, 238] but non of these works overcome all the weaknesses. In PointNet++ [233], PointNet layers are integrated in an EDN. This network considers the hierarchical local neighborhoods of the points. The central modules of PointNet++ are sampling, grouping and feature extraction with PointNet. The Farthest Point Sampling (FPS) algorithm is used to detect principal points. The features of the near surrounding points are grouped in every principal point and fed as one unit to the PointNet layer (Figure 25). With these extensions, larger point clouds can be segmented semantically. The same objective but with two different approaches are pursued in [234]. Their first approach uses different sized input levels (area sizes as shown in Figure 26b) and combines features of them. The features generated from the input levels are given in a consolidation unit. The second approach uses a fixed input block size (Figure 26a). The features from different input blocks are fed into a RNN consolidation unit. The information from four input blocks are considered as information 34 2. State of the art sequence and through the RNN, shared features are created that are used for the semantic segmentation [234]. Figure 25: Structure and functions PointNet++. Encoding of point features in an iterative process considering the local neighborhood: (1) Selecting npoints that are maximal wide away from each other. (2) Grouping of the features in the neighborhoods. (3) Applying a PointNet layer to feature extraction. (4) Repeating this process. Decoding by joint and step-wise interpolation of the features. (5) Classification layer at the end. Figure from [233]. Figure 26: Two methods for creating neighborhood input blocks. a) Fixed block size with the blocks sharing features via RNN. b) Variable block size with different fixed or dynamic radii. Used in CNN or MLP architectures. Inspired by [233, 234]. Similar enhancement is described in [50]. Local multi-scale neighborhoods are implemented by a pyramid pooling function and information is distributed across the network via a RNN layer. This enables learning from the coherence of objects. The enhancements of input features and introducing Conv layer instead of MLP are described in [236]. Self-organizing 35 2. State of the art (SO) networks that use SO maps as inputs and PointNet as encoders are developed in [235]. A feature network consisting of concatenated feature modules and a PointNet network was developed by [237]. In the feature module, local features are formed by the kNN and global feature by the K-means. Furthermore, a centroied loss feature is introduced. [238] use the PointNet network architecture but sub-sample the point cloud by grid in advance and postconnect a CRF operator for fine segmentation. The PointNet++ is also the basis for many enhancements. PointNet++ is based on a standard EDN, which additionally processes features of different abstraction levels (hierarchically). A notable characteristic of PointNet++ is the use of the MLP as a central module. The enhancements of PointNet++ aim on using the local neighborhood of the points as a feature. In order to describe the local orientation of the points, [239] add a scale-invariantfeature-transformation (SIFT) into the PointNet++ as an intermediate layer. They refer this layer as PointSIFT, which describes the orientation of geometric features on different levels of abstraction. A Local Spatial Aware (LSA) layer has been developed by [240]. This can be used to model the feature distribution within an input set as a function. They construct the LSANet for this layer, implementing PointNet++ for local geometric feature extraction. The ShufflePointNet is developed by [241]. Once again, a new layer is implemented into an existing PointNet++. This layer consists of a kNN based grouping of the input point set. For each subgroup, independent features are computed using an MLP. The new features are shuffled, concatenated, and fed into the next layer. Some of the MLP layers are replaced by this new layer in PointNet++. In addition to developments directly related to PointNet, other point-based methods have been developed and a selection of some networks is briefly described below. The division of Figure 19, which subdivides RNN-based, point-based CNN and graph-based methods, is followed. RNNs are often used in conjunction with the MLP network of PointNet, as shown previously. Alternatively, RSNet is a network where the entire point cloud is split into multiple views in x, y, and zdirections. Each slice direction is processed independently. Slices are used to create an order in the point cloud. The features of the different slices are given into a RNN layer, sharing some of the information. The extracted features of the slices are aggregated and then the features of all slice directions are used for the point-wise semantic segmentation [242]. Conv layers convolve an ordered geometric neighborhood so that less data objects, such as pixels, carry more and more deep features. A transfer of this approach to individual independent points is targeted for point-based CNNs. Unlike the MLP, a real convolution is performed over a regional feature distribution. In PointCNN [243] this idea is described in detail and a χ-Conv layer is introduced. PointCNN uses an EDN. The coordinates are transformed into the feature space within a small region. This can be done in simple terms by computing the deviations of each point to a principal point. The principal point carries the area features and is combined with other principal points in the next stage. A similar approach is applied to the KPConv layer by [49]. As an initial step, the point cloud is homogenized using a grid-sampling filter. The point spacing is equal and the objects 36 2. State of the art become distinct by the features of the points. For example, points that are not occupied are marked with free or 0and points that are part of an object are marked with the features of the measurement and occupied or 1. The KPConv layer in the basic version is built on spherical neighborhoods in which the correlation coefficients are calculated for all features and all points to the center of the sphere. The correlation coefficients can be multiplied by any weight matrix of the Conv layer such that they can be chained as EDN. ShellNet [244] also uses spherical input regions in which the features are computed. Circles with different radii are used and the features are processed together. PointConv [245] converts the local neighborhood of points into a continuous density function and a weight function that can be processed with Conv layer. The networks A-CNN [246] and Dilated Point Convolutions [247] are developed on point-based CNN in which the selection of points are optimized. ACNN arranges the point cloud by a local projection of the sub-point cloud onto a disk and processes the features of the points with an encoding MLP network [246]. [247] perform point-wise classification with a CNN added by one FC layer and consider different receptive fields in the encryption phase. As an alternative to feature differences, which represent the relationship between points in a point cloud, graphs are widely used. Graphs represent the relationship between data objects (e.g., points) through edges. Edges describe on one hand which data is in a relationship and on the other hand by the edge weights how this relationship looks. Thus edge weights are usually multidimensional vectors. Therefore, graphs take on an ordering role for a wide variety of data (e.g., social networks and point clouds) that are processed and analyzed with ANNs. A general overview of Graph Neural Networks (GNNs) is given in [248], which provides a clustering of the different types of GNNs. Simple graphs with few nodes are used for the joint processing of RGBand D-images in order to use the local depth information, such as in the case of adjacent objects with similar RGB values [233]. GNNs in combination with MLPs have the purpose that point features and local neighborhood features are used together for depth feature extraction. In the Featurebased Graph Convolutional Network, initial point features are extracted and combined by the graph representation. Subsequently, a sub-graph convolutional network is build, which extracts neighborhood-based features. These depth-features are used for a point-wise classification by a FC layer [249]. A parallel feature decoding with graphs and individual points are described in [250]. For this, the individual features between the branches are shared at different hierarchical levels. The Dynamic Capsule Graph network architecture is based on an EN that processes at the input layer independently different feature types, such as eigenvalues, spectral values and coordinates. The features are combined and shared depth features are created by a chain of encapsulated graph Conv layers, which are summarized by pooling layers. The termination layers are MLPs, where a set of described features is generated for each point [51]. Other GNNs introduce an initial weighting of edge weights or features, during the encoding phase, to reinforce for the differentiation power of the relevant features [155]. The EdgeConv layer, representing the local neighborhood, is often implemented in MLP networks to minimize the disadvantages of PointNet. This EdgeConv layer is used for building 37 2. State of the art modeling [251] and in the analysis of ALS point clouds [252]. However, graphs can also be used for pre-segmentation of the full point cloud. In this case, the graphs are used to build sub-segments of those points that have a large feature similarity. These sub-segments can be processed separately in sub-networks [253]. Additionally, the results of these subnetworks can be used for context-based segmentation [254]. 38 3 Connections of research publications This section explains the connections between the four key publications. The connection with the ROs, the applied approaches and the general results are summarized in sections 3.1 to 3.4. The connections of the ROs and the publications are outlined in section 3.5. 3.1 PAPER 0: PCCT: A point cloud classification tool to create 3D training data to adjust and develop 3D ConvNet The key topic of PAPER 0 is the development of a multi-user, browser-based tool for semantic segmentations of 3D point clouds. This tool is named PCCT1and consists of three independent modules. The data is exchanged via a MariaDB database hosted on a web-server. All three modules can be accessed from any computer within the HafenCity University (HCU) campus network without any local installations. The first module uploads new point clouds into the database. During the upload, the point clouds are converted into a 2D image representation. This 2D image representation is used for an automatic semantic segmentation and visualization within the browser-based tool. After the conversion, individual noise pixels are eliminated by filter algorithms and an edge optimization is performed. RG methods are used to create segments based on features, such as RGB and intensity values. Each image shows only one segment. The image is linked to the 3D cartesian coordinates of the corresponding points via a connection in the database. In the second module, these segment images are randomly loaded by the browser tool and the annotators use a drop-down list to select the appropriate class for the displayed segment. Thus, a class is assigned to each image respectively segment. The third module establishes the relationship between the cartesian coordinates and the classified images. Since each image is classified multiple times and by different annotators, a voting procedure is introduced to assign the most likely classes to the points. The point features are enhanced by the semantic classes. The enhanced point clouds are exported in different ASCII formats. Different projection methods and sets of features for segmentation are tested and applied in studies of PAPER 1 to PAPER 3. In addition, individual processing strategies are used for indoor and outdoor datasets, as they have different characteristics. A brief study is carried out to investigate the semantic accuracy of the PCCT. It is shown that the developed tool is suitable for the task, but an optimization of the tool parameters is necessary in order to achieve higher semantic accuracies. The optimization has to be done on the basis of characteristics which are defined in PAPER 2. The usefulness of manual semantically segmented point clouds is demonstrated on the example of an application with PointNet. 1Github repository: https://github.com/eb17/PCCT 39 4. Evaluation of the research results However, the application additionally specifies which semantic objects of the point cloud are determined and how they are geometrically selected. Typically, BB or irregular 3D solids are used in most datasets and tools. A summary of the annotation tools mostly applied to BIM and DL applications are presented in Tables 1 and 2 (section 2.3), as well as in Table 5 of PAPER 2. These tools are application driven with exception of Cloud Compare and Recap. Tools have been developed for one specific problem in connection with a semantic segmentation. Furthermore, these tools are mostly linked to one specific dataset. Datasets without a relation to a specific tool are rare. Some of these datasets are Semantic3d.net [40] and TUM-MLS-2016 [153], which were created by Cloud Compare. All tools divide the annotation into two steps. These steps are segmentation and labeling. For the most tools, labeling is a simple assignment of a class or an encoded class value to a previously created segment. This step is usually not automated. The more complicated step is segmentation. In the manual segmentation according to semantic aspects, the exact area must be shown in which an object can be separated unambiguously from the environment. This is particularly challenging if very large and very small objects occur. In addition, the object space must not be too large, so that a fluent navigation and visualization through a detailed point cloud is possible. In order to make this possible, the point cloud is usually divided into smaller sections in advanced, based on recording locations, rooms or distance intervals. Due to the complexity of the segmentation according to semantic requirements, errors often occur, so this step is topic for automation. The different methods for automatic segmentation and visualization are explained in section 2.5, as well as in PAPER 0 and PAPER 2. The automation of the segmentation does not necessarily lead to more accurate semantic segments, but only to the fact that these are always determined the same. The determination and selection of the HPs for these segmentation methods is the central adjustment screw in the method development. The selection of the HPs is a very time-consuming task and must be carried out and checked for each individual dataset. DL approaches that only use geometric features are not published, even if e.g. eigenvalues and geometric parameters are suitable for this purpose, as shown by [184] for ML applications. More common semantic segmentation approaches are based on features, such as color and intensity values, since most datasets are created by RGB-D cameras and simple LIDAR scanners (Tables 2 and 3 of PAPER 2). These methods are also described in more detail in section 2.5 and PAPER 0. The accuracy, reliability and efficiency are characteristics that can be used to compare different methods regarding a certain application. These characteristics can only be determined with effort and always for a specific dataset. Usually, third-party tools do not report these characteristics. In order to determine the accuracy, a reference point cloud showing the same scene and with a higher accuracy must be available. The simplest way to create such a dataset is to use synthetic point clouds derived from models [35, 166, 260]. Alternatively, a dataset can be created with a higher accurate and handheld measuring system as described in PAPER 2. In the considered case, a handheld scanner is used to create object-by-object segments. The combination of a very accurate recording system and semantic segmen46 4. Evaluation of the research results tation in the field results in a reliable and accurate semantic point cloud. The creation is labor-intensive in the field and can usually only be applied to small point clouds, due to the usage of special measuring systems. If such a point cloud is available, a geometry comparison can be done to determine incorrectly segmented points. In order to determine the efficiency of annotation tools, the required time must be put in relation to the achievable accuracy of the semantic segmentation. The costs for hardware, software and energy are mostly negligible, since the work of humans labor time causes the highest costs. Very few datasets or tools [41, 53, 54] indicate how long the semantic segmentation takes. Unfortunately, this information is given usually only for one annotator or a small group of annotators as discussed in RQ 1.4 (section 4.1.4). Caused by this missing information the efficiency can not be determined for most datasets. Reliability is determined by multiple independent annotations and compassion with reference data. This does not require a GT dataset, but for comparability of tools, the same dataset should always be used. In addition, reliability in this definition also includes usability and describes how a certain group of humans solves the semantic segmentation task. Thus, aspects such as task comprehension and motivation can be included in this characteristic. For more details to this aspect see RQ 1.4 in section 4.1.4. Conclusion and outlook: The available tools are highly specialized and not suitable for multi-disciplinary applications. Many innovative technical solutions are presented in the literature, but these require different input formats and lead to different semantic representations. Two basic functions, segmentation and classification, are available in most tools. These functions affect the quality of the semantic point cloud, these are discussed in RQs 1.2, 1.3 and 1.4. Meta data is incompletely obtainable for many datasets and annotation tools, and a comparison of them is often not possible. The evaluation of point cloud datasets and annotation tools is addressed in RO 2. This analysis is very time-consuming, but necessary for a better understanding of point cloud data and algorithms, as seen from the first investigations to determine the three most discussed characteristics. 4.1.2 Annotation tools for indoor terrestrial laser scanning point clouds RQ 1.2: Which annotation tools can be used for the semantic segmentation of challenging real-world indoor TLS point clouds? Methodology: Annotation tools and processes were selected based on the literature review explained in the answer of RQ 1.1 in section 4.1.1. The tools SemanticKITTI [53], PC-Annotate [55], Recap [134], and Cloud Compare processes from TUM-MLS-2016 [153] and Semantic3d.net [40] are tested with the reference dataset published in PAPER 2. In addition, the PCCT developed in PAPER 0 is evaluated to investigate its quality. In a pilot study, all tools are evaluated by two volunteers in terms of usability (data format, availability of the software, technical requirements, approximate processing time). In the main study, ten volunteers are asked to perform semantic segmentations with Recap and PCCT (pub47 4. Evaluation of the research results lished in PAPER 2). In a survey, prior knowledge, metrics (e.g., processing time), impression of usability, expected accuracy, and desired changes are asked. The survey is evaluated together with the results of the semantic segmentation. Findings: The results of the pilot study shows that SemanticKITTI and PC-Annotate are not suitable for semantic segmentation of challenging real-world indoor TLS point clouds. The annotation tool of SemanticKITTI is optimized for mobile recorded input data and for dynamicchanging environments. For a semantic segmentation of TLS point clouds, a pseudo navigation file would be needed in order to use this tool. The PC-Annotate tool has a limited selection of classes and a semantic segmentation is only efficiently possible via the fit of regular geometries, which leads to inaccurate segmentations for detailed indoor scenes. The processes of [40, 153] are complex and require a solid knowledge of Cloud Compare, which cannot be assumed for all potential annotators. Therefore, these tools were excluded from the main study. The main study is presented in section 4 of PAPER 2. All quality parameters of the model from section 4.2 and in section 3 of PAPER 2 are determined, so that among other parameters the accuracy and the efficiency are evaluated. The study results demonstrate that the most accurate semantic segmentation is preformed by annotation tools with a free choice of the perspective and a lasso function for segmentation. The annotators work very detail-oriented, which leads to an extended processing time and a decrease in efficiency. The effectiveness is higher with a tools such as Recap. In general, the PCCT, which only allows the annotator to classify, is very efficient, but for very small objects it is not effective. Measurement errors in point clouds make the segmentation difficult for any tool, because boundaries between an object and points representing mix-pixel errors, defuse reflection and comet tails cannot be clearly identified. Geometry-based filtering can make manual and automatic semantic segmentation effective, reliable, and accurate. The segmentation of a coarse pre-segmentation of rooms allows smoother navigation through the point clouds. Current hardware reach its limits for processing very large point clouds with such tools, because the working memories are not large enough. Conclusion and outlook: The research of the available annotation tools shows that only few tools are suitable for the application of modeling indoor rooms. To the best of the author’s knowledge, a systematic evaluation of these tools has been carried out in PAPER 2 for the first time. An annotation tool that can be used across various disciplines is urgently needed. A basis for this development can be the PCCT. Point cloud annotation tools from commercial service providers and CWS are not investigated in detail due to the lack of transparency regarding costs, data security, data rights, and working conditions of crowd-workers. After all, the commercial tools are the drivers for many applications in which semantic point clouds are needed. 48 4. Evaluation of the research results 4.1.3 Development of an annotation tool RQ 1.3: How can semantic segmentation tools for point clouds be enhanced and improved? Methodology: At the beginning of this research1, few scientific tools for semantic point cloud segmentation were available. Commercial service providers predominantly offered semantic segmentation for image data. Commercial and open-source offline software for point cloud processing, such as Geomagic Wrap [261], Cloud Compare, and extensions for CAD programs, are the state of the art. An adaptation of existing systems to improve them is not technically purposeful, therefore the complete development of the PCCT is necessary. A concept for data management, implementation of segmentation methods, classification and visualization is developed on the basis of the analyzed annotation tools from RQ 1.1 in section 4.1.1. The PCCT is iteratively developed and evaluated in studies that are explained in the RQs 1.4, 2.1, 2.2 and 2.3. Findings: The annotation tools available on the market show potential for further developments regarding issues such as data security, capability of a dataset for multiple-users, segmentation and classification functions, and automation of time-consuming sub-operation steps (Figure 28). These issues are considered during the development of the PCCT. The PCCT is an experimental tool, which can be used to optimize the issues and evaluate the experiment. Figure 28: Central issues for improvement in available point cloud annotation tools: Data security, multi-user-capability, segmentation and classification functions, and automation of sub-operation steps. Point clouds are detailed representations of real buildings and make hidden information visible. This information must be kept safe to third-party access for critical infrastructures, such as for port facilities, utility lines, airports, prisons, railroad facilities, or research facilities. 1In 2017 49 4. Evaluation of the research results The usage of CWSs is usually not possible for these infrastructures [55]. Processing of a large dataset by only one annotator is also in most cases not possible and can also lead to classification bias in the semantic point cloud. Therefore, it is necessary to store the data in such a way that it can be accessed in parallel. In the best case, only parts of the point cloud are made available to the user, allowing to edit but preventing understanding the entire infrastructure. The classification bias can be minimized by having different annotators to perform the semantic segmentation. Based on these considerations, a database-based concept is developed for the PCCT, as published in PAPER 0. There, a copy of individual point cloud sections are provided for processing via a browser-based tool. The results of different annotators and classification passes are connected to the original point cloud, but a final assignment of the semantics to a point is made after max-voting over all classifications. The class definition of the semantic segmentation must be transferable into the annotation tool. In order to use the annotation tool in almost any application the list of possible classes • must be re-defined in each case. • include very general classes, cover a large amount of classes. • has a hierarchical organization. A customizable list of classes in the tool can lead to the fact that the class definition is no longer unique. A very general list limits the application scope and a list with too many classes can no longer be overlooked, which can lead to different understandings by the individual annotators. For example, if the class Wall and Facade are available, it is not always clear how they differ. For the PCCT, it is experimented with a class definition that is as general as possible and specialized for building parts and furnitures. Even with this list, a confusion can be seen in the study results due different understanding of classes. The hierarchical organization of the classes using e.g. the WordNet scheme [262] is a technique, which allows a maximum of variety and uniqueness. The technical implementation and the usability of this variant is complex, since an effective navigation must be applied for a list of several thousand words. Currently, list of classes that can be created by an administrator are most effective for practical applications. Some approaches of automation for segmentation of point clouds are described in section 2.3. The special characteristic of TLS point clouds is that they are recorded from a fixed point of view and usually represent a 360° view of the scene. A transformation of the cartesian coordinates into polar coordinates is directly possible. The fix angular increments of the polar coordinates allow a transformation of the 3D point cloud to a structured 2D image. The distance measurements can become a feature variable. Graph-based methods in 2D applications, such as [95, 263], have a high degree of development and provide unique segments. Moreover, 2D segments can usually be visualized and interpreted better by humans than 3D segments. Other developments use BEV approaches of [4, 27] for a semantic segmentation of buildings after the removal of ceilings and floors. These approaches use a floor by floor representation of the building. A disadvantage of the 2D projection is that the geometrical depth is usually not considered in the segmentation step and a distortion oc50 4. Evaluation of the research results curs. Different radii and projections have been tested in the PCCT to perform segmentation automatically and accurately ([264] as well as in PAPER 0 and PAPER 2). Conclusion and outlook: The development of an experimental prototype annotation tool for semantic segmentations is implemented with the PCCT. The PCCT is based on a transformation of the 3D points to 2D pixel, which is partially disadvantageous for some of the applications. Different influencing variables can be tested with the modular-structured PCCT. An annotation tool for any kind of applications and that fits to all requirements from above is not available on the market yet. The optimization of annotation tools is still important but an under-researched topic for more accurate and reliable semantic point clouds. 4.1.4 Human factor in semantic segmentations of point clouds RQ 1.4: How to become a good annotator for semantic point clouds? How can the performance of annotators be measured? What do annotators need and how can the tool support them? Methodology: The influence of the human annotator is a part of the study in PAPER 2 and is determined for the PCCT and Recap. Quantitative parameters such as processing time, precision, and accuracies are measured or calculated by means of the high-quality reference point cloud dataset. Qualitative characteristics are assessed by a questionnaire that is answered before, during, and after the task by the volunteer annotators. Additionally,it is asked for a self-assessment, previous experience and a descriptions of how the tools were used. Findings: The fact that the annotator has a key role for the quality and usability of a semantic point cloud has been noted by [40, 55]. [54] developed valuable rules of thumb for the selection and training of annotators. Feedback during the annotation is given by feedback function in the tool of [265]. Humans are very good at recognizing varying shapes of objects [266]. Following these ideas, a study is conducted, whose results describe the human influence. Guidelines for processes and tool developments should result from this. The ten volunteers of the study had no, little, medium or very much experience in the handling of point clouds and the semantic enhancement of point clouds in advance. This previous knowledge allows an evaluation of annotators training, developed execution process and tool functions. Training documents are prepared for each investigated tool and are given to the volunteers in advance. These documents are assessed and the volunteers have to paraphrase the task in their own words. By this first task, it could be determined that illustrations contribute to a better understanding of the task. The ideas of the application and of point clouds as well as its interpretation variate strongly between the annotators. The given feedback is used to improve the training documents with example images and detailed class descriptions. Helpful in the class description is to clearly include or exclude objects that are geometrically or semantically similar to others. As an example: A door consists of the frame, the leaf and the handle, but not of the window next to the frame. Different volunteers have examined the documents at different stages of the development before the finale experiment. 51 4. Evaluation of the research results The training and the feedback process are planned in detail based on review of the literature. A few days before the experiments, all documents (task description, class definition and illustrated instructions) are given to the volunteers. In order to get familiarized with the task and to answer the first part of the questionnaire. Before the annotation experiment, the tools are explained and questions could be asked. The annotation is done without any supervision. All volunteers are able to solve the tasks. The average class accuracy for most annotators is above 90% (recall and precision). Large differences can be observed among the different classes, such as Floor and Ceiling are above 95% for the parameters recall and precision. The infrequent classes Chair and Table are less accurate and usually vary between 85% and 95% for recall and precision. The percentage of TP points in the class Erroneous points is usually lower than 50% (precision), because in case of doubt object points usually become erroneous points. Large variation is found in the time required. Some volunteers finish within less than 9% of the maximum time. Large differences in time are also found between the tools. The PCCT is more efficient than Recap, but the results are not as correct and the simplicity of use is perceived as tedious. The usage of Recap is complicated at the beginning for some volunteers, so errors occurred more frequently due to the segmentation functions and the processing takes a long time. Point density and variety in the activity are seen as particularly important, in addition to a clear task description. The navigation through the point clouds and self-dependent segmentation, such as using a lasso, are functions that make this possible. A relation between high efficiency and correctness cannot be found. The volunteers find it helpful to be able to ask questions during the task, since there are occasional misunderstandings or ambiguities. These results are taken from the study in PAPER 2. Conclusion and outlook: A good annotator does not need to be an expert in point clouds. Unique task descriptions, class definitions, and continual feedback are most critical for a successful semantic segmentation. Exclusive classification tools, such as PCCT, level the entry threshold, but leads to tiring with very large datasets. 4.2 Development of a quality model for heuristically describing semantic point clouds The findings to the second RO makes the quality of a semantic point cloud measurable. For this purpose, the characteristics of the point cloud are investigated (section 4.2.1) and a quality model is developed (section 4.2.2). In order to use the quality model, it is transformed into an evaluation matrix with which semantic point cloud datasets, annotation tools and automatic workflows can be evaluated (section 4.2.3). 52 4. Evaluation of the research results 4.2.1 Point cloud quality RQ 2.1: What are suitable semantic point clouds? What are the characteristics of point clouds? How can the characteristics of the point cloud be determined, measured and compared? Methodology: A literature research of the characteristics of point clouds has been performed and the creation process of a manual and automatic semantic segmented point clouds has been analyzed on the HCU main building dataset. Findings: The definition for a suitable semantic point cloud, which is developed as a result of this work, is given by the following statement. Definition 1: A good semantic point cloud has a homogeneous density, is free from data gaps, measurement and registration errors, the geometry of the semantic segments corresponds to the objects in reality and the labels accurately describe the object semantics. A semantic point cloud that completely fulfills definition 1 usually does not exist, so that the degree of individual characteristics are determined. This is necessary because point clouds with low quality levels are not suitable for some applications as discussed in RQ 2.2 (section 4.2.2). Before the quality can be determined, the creation process and the characteristics of the semantic point cloud, as well as a definition of errors must be stated: Definition 2: Errors are the influences that lead to not fulfilling the definition for a suitable semantic point cloud. The creation process of semantic point clouds usually consist of the recording, the registration and a subsequent segmentation according to semantic properties of represented objects. The geometric correctness with respect to the recorded surface is the most studied characteristic for the recording and registration [81, 82, 83]. The parameters standard deviation and deviation from a target geometry are typically used to describe the characteristic geometric correctness. Semantic segmentation is the creation of an abstract model from the data model (measured values) and the knowledge about the real world. Such a process is defined in [267] and shown in Figure 29. As additional information the knowledge of a human annotator or the DL algorithms is used to semantically enhance the point cloud. The accuracy of the semantic segmentation is described by the performance of the selected manual or automatic algorithm. This is usually expressed in terms of a ratio of incorrect and correct data objects (pixels and points). The parameters precision,recall or IoU are commonly used. The geometric shape is not expressed by these parameters. Varying error descriptions are found in publications about semantic point clouds. Measurement errors occur in the form of interference points and noise around a surface, which are analyzed for the recording system. In the process of semantic segmentation, these points become an additional semantic class. They are no longer errors for the semantic segmentation. Errors in semantic segmentation are points that are assigned to the wrong class. The definition of errors change in the two-step process. However, it is necessary to consider the 53 4. Evaluation of the research results errors from the previous step in the following one. In the final point cloud, the geometry and semantics of the point cloud should be correctly represented. The recording and registration accuracy are usually not used in semantic segmentation. They are relevant for deriving geometries and models from the semantic point cloud, so that the objects are not distorted and show the geometry. Figure 29: Process of semantic segmentation of point clouds serving as an abstract model of the reality. Taken from PAPER 2 and adapted. For an evaluation and usage of the point clouds, not only just discussed characteristics geometric and semantic precision, as well as correctness are relevant. Other characteristics are: • the availability of data and metadata, • the process reliability, • the completeness of data and processing, • and the consistency of data content (e.g., type of variables), as explained and developed in PAPER 2. In order to determine and compare the characteristics, the developed quality model is an effective tool. For each characteristic, qualitative and quantitative parameters are defined. The development of the set of quality parameters is described in more detail in PAPER 2 and is discussed in RQ 2.2 (section 4.2.2). Thus the comparison of different semantic point clouds is possible. In order to determine the degree of quality of the point cloud for an application, threshold values must be defined for each quality parameter. Conclusion and outlook: The quality of a semantic point cloud is determined by different steps, which are usually concatenated. The definition of what is an error changes from step to step. In order to consider all errors in the final semantic point cloud, these or a description 54 4. Evaluation of the research results of these must be passed on at each step. The quality of a semantic point cloud consists of many different characteristics, which are determined in the individual stages. Structuring them into a quality model developed specifically for semantic point clouds is effective and implemented. 4.2.2 Quality model for point clouds RQ 2.2: How is a quality model for semantic point clouds designed? Which parameters are necessary for the description of the characteristics? Does the quality parameters differ for annotation and automatic semantic segmentation? Methodology: The quality model of [92] is used as the basis of the quality model for semantic point clouds. The characteristics from RQ 2.1 (section 4.2.1) represent the structure of the quality model. In the course of the annotation process the descriptive quality parameters are determined and evaluated. Findings: The developed quality model describes seven characteristics of the point cloud which are directly or indirectly related to the semantic segmentation. Direct related characteristics are accuracy and precision. For example, indirect characteristics are usage constraints, such as for the datasets of [53, 64, 226, 268] which are only allowed to be used as a benchmark. Figure 30: Quality model for semantic enhanced point clouds. Seven relevant characteristics with descriptive quality parameters are shown. Classification of necessary parameters for: Manual segmentations (filled blue circles), manual training data generation (unfilled blue circles) and automatic semantic segmentation (filled green circles). The characteristics become measurable and comparable by quality parameters. The quality parameters of the developed model are summarized in Figure 30 with the respective characteristics. Some quality parameters are in a relation to others, so that e.g. the precision 55 4. Evaluation of the research results 4.3.3 Data pre-processing and data influence RQ 3.3: How can the influence of the dataset be controlled by data-based hyperparameters in the semantic segmentation of point clouds? What are the main influences? Methodology: The dataset, whose characteristics are described and controlled by the DHPs, is an important influencing variable for semantic segmentation of point clouds. The characteristics, the class distribution, the class definition and the incorrectly measured points are analyzed and evaluated by empirical investigations with the developed workflow of PAPER 1 and PAPER 3. Findings: The DHPs are set by the dataset. Examples are the available feature variables, the class distribution or the size of the dataset. A collection of the most common DHPs is summarized in Fig 33. An overview of the structure of the dataset, the semantic content and the features allow a systematization of the influences and its investigation. The semantic DHPs have been investigated in RO 2 (sections 4.2.2 and 4.2.3). The selection and local computation of geometric and spectral features are studied for general ML methods in [39, 273]. In addition, the dataset size, the normalization of features, and the density of the point clouds are considered in many network architecture developments [37, 200, 274]. Rules of thumb can be derived from this, but they are not supported by any systematic proof. The structure of point cloud datasets is usually only investigated with respect to the input formats. With few exceptions [172, 250, 271, 275, 276] class definitions, characteristics of erroneous points, and class size differences are not considered in point cloud datasets, even though these are considered to be a well-known influencing factor in semantic segmentation of images [258, 277]. These three DHPs are explored in PAPER 1 and PAPER 3. Figure 33: DHPs for semantic point clouds. The DHPs can be distinguished according to structural, semantic, geometric and spectral characteristics. A selection of the most common DHPs for each property is summarized. The class Erroneous points is usually determined less precisely by most algorithms for semantic point cloud segmentations than the object classes. This is shown by the analysis of point-based CNN at the leader board of the TLS dataset of Semantic3D.net [40] (Figure 34). In addition, this analysis shows that more frequent classes, such as Building and Road, are determined more accurately than the smaller class Tree. 62 4. Evaluation of the research results Figure 34: Semantic segmentation accuracy (IoU) of four common network architectures for the dataset: Semantic3d.net [40]. Selection of four from eight classes of this dataset. The class Scanning Artifacts, which is equal to the class Erroneous point, can be detected poorly compared to the larger classes. Values are taken from the leader board of [40]. The observations indicate that such influences exist (Figure 34). In PAPER 1, the influence of the presence or absence of the class Erroneous points are investigated. Figure 35a shows the results of the semantic segmentation without Erroneous points. The frequent classes Tree and Building are determined with more than 80% recall and precision. The infrequent class Street Furniture is determined with less than 10% recall and precision. If the class Erroneous points is added, recall and precision for all classes are lower than 54%, as shown in Figure 35b. From this example, it can be seen that there is an influence of the erroneous points in the semantic segmentation of point cloud datasets. Erroneous points are arranged similarly as object points, as erroneous points are caused by multiple and diffuse reflections. In the larger study of PAPER 2, the influence could be confirmed. However, with a large indoor datasets the influence is less. For infrequent class a positive effect of the presence of the class Erroneous points can also be observed by a higher precision value. Figure 35: Comparison of semantic accuracy (recall and precision) on the point cloud of the HafenCity (outdoor) dataset: a) Without the class Erroneous points and b) With the class Erroneous points. Selection of three classes that have different frequencies in the dataset. Data from PAPER 1. 63 4. Evaluation of the research results The influence of the class Erroneous points is therefore not the only crucial factor for the semantic accuracy, but the combination of the semantic classes and its point distribution. The division according to classes takes place on the basis of the class definition, that rules which classes are determined with the semantic segmentation. The best possible differentiation is always possible if the features of the point clouds can be clearly separated from each other. The ceiling and the floor can be well separated by different values for the feature variable height. Such considerations can be taken into account when developing the class definition. If, due to the task, a separation by classes with very similar features is not possible, a stepwise semantic segmentation can be performed as outlined in Figure 36. Similar classes are combined in a super class in the first stage (Network A) and then Network B is used for the separation. The influence of a class definition and the hierarchical process could be demonstrated in the study of PAPER 2. This shows a slight increase in semantic accuracy for the Window and Door classes. However, this developed process is strongly linked to the individual rooms. Figure 36: Step-wise semantic segmentation for improved differentiation of classes with similar features. With network A, a segmentation is performed for general classes, which is refined in network B. An adjustment of the class definition does not necessarily lead to the classes having the same number of points. Classes such as Wall,Floor and Ceiling are more frequently represented classes in the point cloud, than Doors,Furniture and Windows due to their larger surfaces. The learning algorithm will learn these classes more often than the infrequent ones due to the more frequent feeding with points whose class is wall,floor or ceiling. To enhance learning in favor of the infrequent classes, their proportion can be artificially increased (Figure 37a), the inputs can be emphasized with a higher proportion of infrequent points (Figure 37c), or in a loss calculation, the points of the infrequent classes can be rewarded by a higher weight (Figure 37b). Extensions of points can be done randomly or by considering local conditions, as with the SMOTE method. These three approaches are investigated in PAPER 3 in several variants using the general HP set from RQ 3.2 (section 4.3.2). Again, a modest and scenedependent increase in semantic accuracy is observed due to a higher recall for the infrequent classes. 64 4. Evaluation of the research results Figure 37: Dataset optimization methods for semantic point cloud segmentation: a) Dataset expansion by randomly copying points, b) weighting the loss function, and c) dataset expansion by copying inputs with infrequent points. Conclusion and outlook: DHPs represent a measurable influence for the semantic segmentation of point clouds. The DHPs: Class definitions, proportion of erroneous points and class size differences influence the semantic segmentation results and are optimized in the context of this work. However, this optimization can only be valid for a proportion of the scenes. Further investigations on DHPs are necessary to establish rules for the optimal choice of them. An analysis of the semantics in the scenes is necessary. 65 5 Conclusion and outlook This section summarizes the key findings, the conclusions and the responses to the RQs (section 5.1). Intermediate conclusions are summarized in section 4 at the end of each RQ. Next steps for further development and optimization based on the results of this work are described in section 5.2. 5.1 Conclusion This thesis shows that the development of a workflow with which any type of point clouds can be semantically augmented, in any kind of application is not possible at the current state of the art. The key reasons are lack of knowledge about datasets and undefined rules for HPs. Nevertheless, DL methods are most suitable for semantic segmentation. The semantic enhancement of 3D point clouds is a necessary step in order to produce highly accurate digital models of the real world. The semantics of point clouds is central for the usability and the interpretability, if automatic digital processing should or must used. DL algorithms, such as point-based CNN, produce accurate semantic point clouds if optimal HPs and sufficient training point clouds are used. Optimal HPs, algorithms, and training data were explored on the HCU main building dataset and predominantly with the PointNet method in this thesis. In order to investigate the influences three developments were necessary: • The development of a workflow for automatic semantic segmentation. • The development of a tool for manual semantic segmentation. • The development of a quality model for the process and the semantic point cloud. The first challenge in workflow development was, that most DL methods use their own data pre-processing methods. This is dictated by the data format of the recording sensor and is not adapted for the optimal performance of the algorithm or to the data content. The second challenge is the advancement of the hardware, APIs, and DL methods. To effectively consider new hardware and API developments as well as different DL algorithms, a modular workflow which is independent of the dataset formats has been developed. This workflow consists of the modules for feature extension, feature value normalization, input formatting, training, evaluation and application. Each module can be modified by a few parameters. The described workflow is a bridge between high-end developments and practical measurements. Tools for point cloud annotations shall accelerate, simplify, standardize and optimize the very labor-intensive, individual process of point cloud annotation. The semantic point clouds are crucial, because they are the knowledge carrier of DL algorithms. The development of the PCCT is based on the above requirements and reduces individual human influences by 66 5. Conclusion and outlook means of automatic segmentation. A reduction of the processing time compared to Recap was possible by 42% on average. However, this resultes in a decrease in semantic accuracy of up to 16% for recall and up to 12% for precision across all classes. The PCCT multi-user capability makes it ideal for studies in which different segmentation performances are investigated as an influence or allow efficient processing of large datasets by different users. Which metrics are used for an evaluation is not standardized in literature and the exact expresses of each metric is sometimes not clear. In addition, these metrics usually only represent the semantic accuracy in relation to a GT dataset. All these ambiguities in the definitions limit the meaningfulness of the metrics. A complete evaluation of a semantic point cloud includes several characteristics, such as geometric accuracy, reliability, completeness, availability, and integrity. These characteristics of the semantic point cloud are represented by the quality model developed in this work. A complete evaluation and comparison of the dataset characteristics and performance of all processing steps is thus possible. The metrics are integrated into the quality model as quality parameters and define together with additional quality parameters a higher significance model for systematic investigations, comparisons and the examination for the suitability of data and algorithms in a specific application. The three developments of the thesis are used to study the influence of datasets and point clouds in manual and automatic semantic segmentations. Differences in accuracy, effectiveness, and efficiency in manual semantic segmentation were identified, caused by the functions in the tools, the user training, and the point clouds. It was found that further developments of annotation tools are mandatory in order to produce sufficient training data for productive applications of DL algorithms. Training data point clouds are key materials, but have been rarely studied. This work focuses on the characteristics of the datasets and the point clouds. The presence of erroneous points affects the semantic segmentation by decreasing the semantic accuracy of frequent classes. In contrast, for infrequent classes an increase in semantic accuracy is observed of up to 22% (interior) for the PointNet baseline method, if the class Erroneous points is part of the class definition. Unequal class partitioning leads to the fact that infrequent classes have a lower accuracy. In many examples of this thesis it can be observed that frequent classes are learned very well (>90% recall) and infrequent classes are not learned (<50% recall). Systematic and artificial modification of the point cloud dataset can improve the recognizability of infrequent classes (recall >50%). Classes that show very similar features are more difficult to separate than classes that have different features (e.g., heights). A hierarchical approach for the class definition could in some cases (e.g., windows and doors as openings) improve the semantic segmentation. This can be observed from the recall, which is up to 43% higher for the class Openings in the baseline method. Finally, not only the characteristics of the individual points are crucial, but also the characteristics of a neighborhood. How the local and global neighborhood can be taken into account is discussed in many papers, but general rules that allow to apply them are not available yet. 67 5. Conclusion and outlook Some attempts have been made to take into account different densities and different large areas in the input to the algorithm. However, its influence remains to be investigated in detail. Some possible approaches have been examined in this work and some new approaches will be explained in the outlook (section 5.2). 5.2 Outlook The research of this dissertation reveals three additional research areas that require further investigations and developments. These research areas are the optimization of the input to DL algorithms (section 5.2.1), the manual pre-selection of feature variables (section 5.2.2), and the application of the quality model (section 5.2.3). Furthermore, research on algorithms, on strategies for optimization of HPs and combinations of ML and DL are the current issues. 5.2.1 Input format In the semantic segmentation of point clouds with DL methods, the semantics are learned from the arrangement of the points and its additional feature variables, such as intensity, point normals or color values. Point clouds containing hundreds of thousands of points cannot be fed into a network architecture at once, so only a subset can be processed at each time. As a result, a subset of the information can be used for global feature extraction and classification. Information from the entire point cloud is not known at all (in the case of PointNet) or only insufficiently known (in the case of RandLaNet or PointNet++). For PointNet and 2D CNNs a possible approach to address this issue is the use of graphs as input format, such as described in section 2.6.2. Supplementing these methods of the literature, local and global adjacency matrices expressing the adjacency of the points can be computed from a kNN graph for each point. Adjacency matrices have the advantage that they order the topological relationships and can be represented in a 2D format. By multiplying the adjacency matrices with the features of the points, a tensor can be computed for input to a 2D CNN, such as UNet (Figure 38). Also, the adjacency matrix for a local area may be used as a direct input to PointNet and is a carrier of additional information about the local relationships of the points. Figure 38: Process for creating an adjacency matrix and applying it as a network input. 68 5. Conclusion and outlook Initial tests of this method show that an increase in semantic accuracy is possible with a PointNet architecture. Currently, data preparation is the bottleneck of this method. Larger studies with different datasets need to be conducted to validate this observation. The input format of the point cloud to the algorithm is seen as an important influence that must be investigated systematically in future research. 5.2.2 Hand-crafted feature selection The manual selection of features for semantic segmentation with ML methods is necessary for pre-processing of point cloud data, which was investigated and optimized in [185]. This idea is applied to CNN in [40, 278] by computing moments and eigenvalues as additional features. Subsequently, an optimization of the set of input features is performed. Eigenvalues and moments carry information not only about the point itself, but also about the neighborhood, so they bring in more global information into the algorithm. To compute this type of features, it is necessary to define a local neighborhood over which the eigenvalues are determined. In a test, the sum of eigenvalue, the planarity and the linearity are calculated, using a radius of 3.5 cm for including the neighborhood. These differences of values are shown in Figures 39b to 39d. Figure 39a shows the GT classification of the point cloud. It can be seen that object boundaries can be distinguished more accurate than in the case of most spectral features. Figure 39: Eigenvalue based features calculated from geometric features (x, y, z): a) GT semantic segmentation. b) Sum of eigenvalues as feature. c) Planarity as a feature. d) Linearity as a feature. A histogram is shown next to the legend. In the experiment for this approach two tests were performed with the PointNet workflow. These experiments show that for the class combination Erroneous points and Objects com69 5. Conclusion and outlook parable accuracies are achieved with baseline method (Figure 40). For the class combination of Interior and Building parts, it can be demonstrate that with the new eigenvalue feature set recall and precision decrease of more than 30% (Figure 41). In further studies the feature calculation and the variable selection have to be optimized to improve automatic semantic segmentation. Figure 40: Semantic point cloud for the classes Objects and Erroneous points. The semantic segmentation is performed using the PointNet-based workflow with the features: x-, y-,z-coordinates, sum of eigenvalues,planarity and linearity. Figure 41: Semantic point cloud for the classes Building parts and Interior. The semantic segmentation is performed using the PointNet-based workflow with the features: x-, y-, z-coordinates, sum of eigenvalues,planarity and linearity. 5.2.3 Point cloud quality assessment The evaluation of semantic segmentation in real-world applications has been treated only marginally in the developments so far. Since the influence of the data and its enhancement is crucial for the performance of the algorithms, these should be investigated in more detail. The developed quality model of this work provides a basis for this, for which thresholds per quality parameter still have to be defined. These should be primary for frequent applications of semantic segmentation, as it is the case for indoor scenes, facades, or street scenes. In addition, these thresholds must be algorithm-dependently determined. 70 Bibliography [1] P. Welchering. Nach der Corona-Pandemie - Wie die Digitalisierung der Arbeitswelt jetzt weitergeht. Ed. by Deutschlandfunk. 2021. URL: https://www.deutschlandfunk. de/nach-dercorona-pandemiewie-diedigitalisierung-der100.html (visited on 04/15/2023). [2] H. Kahmen. Angewandte Geodäsie: Vermessungskunde. De Gruyter Verlag, 2005. DOI: 10.1515/9783110911145. [3] A. Wieser, H. Kuhlmann, V. Schwieger, and W. Niemeier. “Ingenieurgeodäsie – eine Einführung”. In: Ingenieurgeodäsie. Ed. by W. Schwarz. Springer Spektrum, Berlin, Heidelberg, 2017, pp. 1–22. DOI: 10.1007/978-3-662-47188-3_19. [4] V. Stojanovic, H. Shoushtari, C. Askar, A. Scheider, C. Schuldt, N. Hellweg, and H. Sternberg. “A Conceptual Digital Twin for 5G Indoor Navigation”. In: The Eleventh International Conference on Mobile Services, Resources and Users - MOBILITY 2021. 2021, pp. 5–14. [5] H. Hosamo, M. H. Hosamo, H. K. Nielsen, P. R. Svennevig, and K. Svidt. “Digital Twin of HVAC system (HVACDT) for multiobjective optimization of energy consumption and thermal comfort based on BIM framework with ANN-MOGA”. In: Advances in Building Energy Research 17.2 (2022), pp. 125–171. DOI: 10.1080/17512549.2022.2136240. [6] R. Kaden, C. Clemen, R. Seuß, J. Blankenbach, R. Becker, A. Eichhorn, A. Donaubauer, and U. Gruber. Leitfaden Geodäsie und BIM. Tech. rep. 2.1. DVW e.V. und Runder Tisch GIS e.V., 2020. URL: https : / / dvw . de / images / anhang / 2757 / leitfaden-geodaesie-und-bim2020onlineversion.pdf (visited on 07/01/2022). [7] T. Bender, M. Härtig, E. Jaspers, M. Krämer, M. May, M. Schlundt, and N. Turianskyj. “Building Information Modeling”. In: CAFM-Handbuch. Springer Fachmedien Wiesbaden, 2018. Chap. 11, pp. 295–324. DOI: 10.1007/978-3-658-21357-2_11. [8] BIM.Hamburg. BIM-Leitfaden für die FHH Hamburg. Tech. rep. BIM.Hamburg, 2019. [9] DIN 18710. Engineering survey. Deutsche Norm, 2010. [10] R. Becker, E. Lublasser, J. Martens, R. Wollenberg, H. Zhang, S. Brell-Cokcan, and J. Blankenbach. “Enabling BIM for Property Management of Existing Buildings Based on Automated As-is Capturing”. In: Proceedings of the International Symposium on Automation and Robotics in Construction. International Association for Automation and Robotics in Construction, 2019. DOI: 10.22260/isarc2019/0028. [11] K. Soliman, K. Naji, M. Gunduz, O. B. Tokdemir, F. Faqih, and T. Zayed. “BIM-based Facility Management Models for Existing Buildings”. In: Journal of Engineering Research (2021). DOI: 10.36909/jer.11433. I Bibliography [75] O. Wasenmüller and D. Stricker. “Comparison of Kinect V1 and V2 Depth Images in Terms of Accuracy and Precision”. In: Computer Vision – ACCV 2016 Workshops. Springer International Publishing, 2017, pp. 34–45. DOI: 10.1007/978-3-319-544274_3. [76] K. Sauermann. “Modellierung von Bestandsbauwerken am Beispiel der Messe Dortmund, Halle 4”. In: 22. Internationale Geodätische Woche Obergurgl. Ed. by T. Weinold. 2023, pp. 67–77. ISBN: 978-3-87907-738-0. [77] C. Wang, S. Hou, C. Wen, Z. Gong, Q. Li, X. Sun, and J. Li. “Semantic Line Framework-Based Indoor Building Modeling Using Backpacked Laser Scanning Point Cloud”. In: ISPRS Journal of Photogrammetry and Remote Sensing 143 (2018), pp. 150–166. DOI: 10.1016/j.isprsjprs.2018.03.025. [78] B. Stahl and A. Reiterer. “Mobile Mapping Platform with Integrated End-To-End Data Processing Chain for Smart City Applications”. In: Remote Sensing Technologies and Applications in Urban Environments VII. Ed. by N. Chrysoulakis, T. Erbertseder, and Y. Zhang. SPIE, 2022. DOI: 10.1117/12.2633965. [79] T. Luhmann, S. Robson, S. Kyle, and J. Boehm. Close-Range Photogrammetry and 3D Imaging. De Gruyter, 2013. DOI: 10.1515/9783110302783. [80] F. Neitzel, B. Gordon, and D. Wujanz. DVW-Merkblatt 7-2014, Verfahren zur standardisierten Überprüfungvon terrestrischen Laserscannern (TLS). Tech. rep. DVW, 2014. URL: https://dvw.de/veroeffentlichungen/standpunkte/1149-verfahren-zurstandardisiertenueberpruefungvonterrestrischenlaserscannerntls (visited on 10/28/2021). [81] D. Wujanz, M. Burger, F. Tschirschwitz, T. Nietzschmann, F. Neitzel, and T. Kersten. “Determination of Intensity-Based Stochastic Models for Terrestrial Laser Scanners Utilising 3D-Point Clouds”. In: Sensors 18.7 (2018), p. 2187. DOI: 10.3390/ s18072187. [82] H. Neuer. “Qualitätsbetrachtungen zu TLS-Daten”. In: DVW-Seminar Qualitätssicherung geodätischer Messund Auswerteverfahren. Vol. 95. DVWArbeitskreis 3 »Messmethoden und Systeme«, 2019, pp. 69–89. [83] T. P. Kersten, M. Lindstaedt, and M. Stange. “Geometrische Genauigkeitsuntersuchungen aktueller terrestrischer Laserscanner im Labor und im Feld”. In: AVN 2.128 (2021), pp. 59–67. [84] B. Ortner, S. Papp, and M. Meir-Huber. “Einleitung”. In: Handbuch Data Science. Carl Hanser Verlag GmbH & Co. KG, 2019, pp. 1–14. DOI: 10.3139/9783446459755.001. [85] T. A. Runkler. “Einführung”. In: Data Mining. Springer Fachmedien Wiesbaden, 2015, pp. 1–3. DOI: 10.1007/978-3-8348-2171-3_1. [86] G. Langs and R. Wazir. “Machine Learning”. In: Handbuch Data Science. Carl Hanser Verlag GmbH & Co. KG, 2019, pp. 177–198. DOI: 10.3139/9783446459755.006. VIII Bibliography [87] M. Meir-Huber, S. Papp, and B. Ortner. “Datenarchitekturen”. In: Handbuch Data Science. Carl Hanser Verlag GmbH & Co. KG, 2019, pp. 103–123. DOI: 10.3139/ 9783446459755.003. [88] T. Dullien. “Maschinelles Lernen und künstliche Intelligenz in der Informationssicherheit”. In: Datenschutz und Datensicherheit 42.10 (2018), pp. 618–622. DOI: 10.1007/ s11623-018-1012-3. [89] R. Wazir and G. Langs. “Statistik-Grundlagen”. In: Handbuch Data Science. Carl Hanser Verlag GmbH & Co. KG, 2019, pp. 141–176. DOI: 10.3139/9783446459755. 005. [90] A. Ng and K. Soo. “Das Wichtigste in Kürze …” In: Data Science – Was ist das eigentlich?! Springer Berlin Heidelberg, 2018, pp. 1–18. DOI: 10.1007/978-3-66256776-0_1. [91] B. Ortner. “Data Pipelines”. In: Handbuch Data Science. Carl Hanser Verlag GmbH & Co. KG, 2019, pp. 125–140. DOI: 10.3139/9783446459755.004. [92] T. Wiltscho. “Sichere Information durch infrastrukturgestützte Fahrerassistenzsysteme zur Steigerung der Verkehrssicherheit an Straßenknotenpunkten”. PhD thesis. University Stuttgart, 2004. [93] T. A. Runkler. “Datenvorverarbeitung”. In: Data Mining. Springer Fachmedien Wiesbaden, 2015, pp. 23–36. DOI: 10.1007/978-3-8348-2171-3_3. [94] M. Weinmann, B. Jutzi, S. Hinz, and C. Mallet. “Semantic Point Cloud Interpretation Based on Optimal Neighborhoods, Relevant Features and Efficient Classifiers”. In: ISPRS Journal of Photogrammetry and Remote Sensing 105 (2015), pp. 286–304. DOI: 10.1016/j.isprsjprs.2015.01.016. [95] P. F. Felzenszwalb and D. P. Huttenlocher. “Efficient Graph-Based Image Segmentation”. In: International Journal of Computer Vision 59.2 (2004), pp. 167–181. DOI: 10.1023/b:visi.0000022288.19776.77. [96] R. Schnabel, R. Wahl, and R. Klein. “Efficient RANSAC and for Point-Cloud and Shape Detection”. In: Computer Graphics Forum 26.2 (2007), pp. 214–226. ISSN: 0167-7055. DOI: 10.1111/j.1467-8659.2007.01016.x. [97] M. A. Fischler and R. C. Bolles. “Random Sample Consensus”. In: Communications of the ACM 24.6 (1981), pp. 381–395. DOI: 10.1145/358669.358692. [98] N. Shukla. Machine Learning with Tensorflow. Ed. by K. Fricklas. manning pub, 2018. 272 pp. ISBN: 1617293873. [99] A. Y. Ng and M. I. Jordan. “On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes.” In: Advances in Neural Information Processing Systems (2002), pp. 841–848. [100] E. Che, J. Jung, and M. Olsen. “Object Recognition, Segmentation, and Classification of Mobile Laser Scanning Point Clouds: A State of the Art Review”. In: Sensors 19.4 (2019), p. 810. DOI: 10.3390/s19040810. IX Bibliography [101] L. I. Smith. A tutorial on Principal Components Analysis. Tech. rep. Report No. OUCS2002-12. University of Otago, NZ, 2002. URL: http://hdl.handle.net/10523/7534 (visited on 02/21/2023). [102] T. A. Runkler. “Datenvisualisierung”. In: Data Mining. Springer Fachmedien Wiesbaden, 2015, pp. 37–58. DOI: 10.1007/978-3-8348-2171-3_4. [103] Y. Xu, L. Hoegner, S. Tuttas, and U. Stilla. “Voxeland Graph-Based Point Cloud Segmentation of 3D Scenes Using Perceptual Grouping Laws”. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences IV-1/W1 (2017), pp. 43–50. DOI: 10.5194/isprs-annals-iv-1-w1-43-2017. [104] R. Triebel, J. Shin, and R. Y. Siegwart. “Segmentation and Unsupervised Part-based Discovery of Repetitive Objects”. In: Robotics: Science and Systems VI. Robotics: Science and Systems Foundation, 2010. DOI: 10.15607/RSS.2010.VI.009. [105] A.-V. Vo, L. Truong-Hong, D. F. Laefer, and M. Bertolotto. “Octree-Based Region Growing for Point Cloud Segmentation”. In: ISPRS Journal of Photogrammetry and Remote Sensing 104 (2015), pp. 88–100. DOI: 10.1016/j.isprsjprs.2015.01.011. [106] M. Ahmed, R. Seraj, and S. M. S. Islam. “The k-means Algorithm: A Comprehensive Survey and Performance Evaluation”. In: Electronics 9.8 (2020), p. 1295. DOI: 10. 3390/electronics9081295. [107] H. Zhang. “The Optimality of Naive Bayes”. In: The Florida AI Research Society. 2004. [108] T. A. Runkler. “Klassifikation”. In: Data Mining. Springer Fachmedien Wiesbaden, 2015, pp. 89–107. DOI: 10.1007/978-3-8348-2171-3_8. [109] J. C. Stoltzfus. “Logistic Regression: A Brief Primer”. In: Academic Emergency Medicine 18.10 (2011), pp. 1099–1104. DOI: 10.1111/j.1553-2712.2011.01185.x. [110] L. Jiang, Z. Cai, D. Wang, and S. Jiang. “Survey of Improving K-Nearest-Neighbor for Classification”. In: IEEE Fourth International Conference on Fuzzy Systems and Knowledge Discovery. IEEE, 2007. DOI: 10.1109/fskd.2007.552. [111] G. Shakhnarovich and T. Darrell. Nearest-Neighbor Methods in Learning and Vision. Theory and Practice (Neural Information Processing). The MIT Press, 2006, p. 262. ISBN: 9780262195478. [112] K. Kunze. Hauptseminar Machine Learning:Support Vector Machines, Kernels. 2004. URL: https://campar.in.tum.de/twiki/pub/Far/MachineLearningWiSe2003/kunze_ ausarbeitung.pdf (visited on 01/12/2023). [113] S. Levine. Week 6: Support Vector Machines. 2016. URL: https : / / courses . cs . washington.edu/courses/cse446/16sp/svm_2.pdf (visited on 01/13/2023). [114] B. DeVille. “Decision Trees”. In: Wiley Interdisciplinary Reviews: Computational Statistics 5.6 (2013), pp. 448–455. DOI: 10.1002/wics.1278. [115] L. Breiman. “Random Forests”. In: Machine Learning 45.1 (2001), pp. 5–32. DOI: 10.1023/a:1010933404324. X Bibliography [116] D. Koguciuk, Ł. Chechliński, and T. El-Gaaly. “3D Object Recognition with Ensemble Learning - A Study of Point Cloud-Based Deep Learning Models”. In: Advances in Visual Computing (2019), pp. 100–114. DOI: 10.1007/978-3-030-33723-0_9. [117] S. Strecker. “Künstliche Neuronale Netze - Aufbau und Funktionsweise”. In: Arbeitspapiere WI. Lehrstuhl für Allg. BWL und Wirtschaftsinformatik, Johannes Gutenberg-Universität Mainz, 1997. [118] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. The MIT Press, 2016. 800 pp. ISBN: 0262035618. URL: https://www.ebook.de/de/product/26337726/ ian_goodfellow_yoshua_bengio_aaron_courville_deep_learning.html (visited on 01/13/2023). [119] T. Rashid. Neuronale Netze selbst programmieren. D Punkt Verlag GmbH, 2017. 232 pp. ISBN: 3960090439. [120] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer. “Efficient Processing of Deep Neural Networks: A Tutorial and Survey”. In: Proceedings of the IEEE 105.12 (2017), pp. 2295–2329. DOI: 10.1109/jproc.2017.2761740. [121] M. Heinert. “Artificial Neural Networks – How to Open the Black Boxes?” In: First Workshop on Application of Artificial Intelligence and Innovations in Engineering Geodesy. 2008, pp. 42–62. [122] I. Goodfellow. Deep Feedforward Networks. 2016. URL: http : / / www . deeplearningbook.org/lecture_slides.html (visited on 01/13/2023). [123] L. Fei-Fei, J. Wu, and R. Gao. CS231n: Convolutional Neural Networks for Visual Recognition. Ed. by Standford vision and Learning Lab. 2020. URL: https://cs231n. github.io/neural-networks-case-study/ (visited on 02/02/2023). [124] D. Maturana and S. Scherer. “VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2015, pp. 922–928. DOI: 10.1109/iros.2015.7353481. [125] N. Höft. “Bildsegmentation in Objekt-Klassen mit Konvolutionalen Neuronalen Netzen”. Bachelorthesis. University Bonn, 2014. [126] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. “Deep Learning for 3D Point Clouds: A Survey”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 43.12 (2019), pp. 4338–4364. DOI: 10.1109/tpami.2020.3005434. [127] A. Krizhevsky, I. Sutskever, and G. E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. In: Communications of the ACM 60.6 (2017), pp. 84– 90. DOI: 10.1145/3065386. [128] O. Ronneberger, P. Fischer, and T. Brox. “U-Net: Convolutional Networks for Biomedical Image Segmentation”. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2015, pp. 234–241. DOI: 10.1007/978-3-319-24574-4_28. XI Bibliography [129] C. Olah. Blog: Understanding LSTM Networks. 2015. URL: https://colah.github.io/ posts/2015-08-Understanding-LSTMs/ (visited on 03/29/2023). [130] D. Passos and P. Mishra. “A Tutorial on Automatic Hyperparameter Tuning of Deep Spectral Modelling for Regression and Classification Tasks”. In: Chemometrics and Intelligent Laboratory Systems 223 (2022), p. 104520. ISSN: 0169-7439. DOI: 10. 1016/j.chemolab.2022.104520. [131] F. Kaufmann, C. Glock, and T. Tschickardt. “ScaleBIM: Introducing a Scalable Modular Framework to Transfer Point Clouds into Semantically Rich Building Information Models”. In: Proceedings of the 2022 European Conference on Computing in Construction. University of Turin, 2022. DOI: 10.35490/ec3.2022.194. [132] T. Hackel. “Large-scale Machine Learning for Point Cloud Processing”. PhD thesis. ETH Zürich, Institute of Geodesy and Photogrammetry, 2018. DOI: 10.3929/ethz-b000264691. [133] W. Zimmer, A. Rangesh, and M. Trivedi. “3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streams”. In: IEEE Intelligent Vehicles Symposium. IEEE, 2019, pp. 1816–1821. DOI: 10.1109/ivs.2019.8814071. [134] Autodesk-Recap. Youtube Channel. 2021. URL: http://https://www.youtube.com/ user/autodeskrecap/ (visited on 06/24/2021). [135] PointCab. Punktwolken Software für alle Bedürfnisse. 2023. URL: https://pointcabsoftware.com (visited on 01/23/2023). [136] J. Simon. Blog: New - Label 3D Point Clouds with Amazon SageMaker Ground Truth. Ed. by A. SageMaker. 2020. URL: https://aws.amazon.com/de/blogs/aws/new-label3d-point-clouds-with-amazon-sagemaker-ground-truth/ (visited on 01/23/2023). [137] basicAI. AI Training Data Solutions for All Industries. 2023. URL: https://www.basic. ai/services (visited on 01/23/2023). [138] scale. Better Data, 2023. URL: https://scale.com/ (visited on 01/23/2023). [139] R. Richter. Wie unsere Plattfrom funktioniert. Ed. by P. C. Technology. 2023. URL: https://www.pointcloudtechnology.com/de/#loesung (visited on 01/23/2023). [140] R. Monica, J. Aleotti, M. Zillich, and M. Vincze. “Multi-label Point Cloud Annotation by Selection of Sparse Control Points”. In: International Conference on 3D Vision. IEEE, 2017. DOI: 10.1109/3dv.2017.00042. [141] P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and G. Ranzuglia. “MeshLab: An Open-Source Mesh Processing Tool”. In: Eurographics Italian Chapter Conference (2008). [142] M. Veit and A. Capobianco. “Go Then Tag: A 3-D Point Cloud Annotation Technique”. In: IEEE Symposium on 3D User Interfaces. IEEE, 2014. DOI: 10.1109/3dui.2014. 6798886. XII Bibliography [143] S. Kulkarni, M. Chandrashekaraiah, and S. Raghunandan. “3D Annotation Tool Using LiDAR”. In: IEEE Global Conference for Advancement in Technology. Bangalore, India: IEEE, 2019, pp. 1–4. ISBN: 978-1-7281-3695-0. DOI: 10.1109/gcat47503.2019. 8978301. [144] B. Wang, V. Wu, B. Wu, and K. Keutzer. “LATTE: Accelerating LiDAR Point Cloud Annotation via Sensor Fusion, One-Click Annotation, and Tracking”. In: IEEE Intelligent Transportation Systems Conference. IEEE, 2019. DOI: 10.1109/itsc.2019.8916980. [145] H. A. Arief, M. Arief, G. Zhang, Z. Liu, M. Bhat, U. G. Indahl, H. Tveite, and D. Zhao. “SAnE: Smart Annotation and Evaluation Tools for Point Cloud Data”. In: IEEE Access 8 (2020), pp. 131848–131858. DOI: 10.1109/access.2020.3009914. [146] D. Coffey, N. Malbraaten, T. B. Le, I. Borazjani, F. Sotiropoulos, A. G. Erdman, and D. F. Keefe. “Interactive Slice WIM: Navigating and Interrogating Volume Data Sets Using a Multisurface, Multitouch VR Interface”. In: IEEE Transactions on Visualization and Computer Graphics 18.10 (2012), pp. 1614–1626. DOI: 10.1109/tvcg.2011.283. [147] N. O’Mahony, S. Campbell, A. Carvalho, L. Krpalkova, D. Riordan, and J. Walsh. “Point Cloud Annotation Methods for 3D Deep Learning”. In: 13th International Conference on Sensing Technology. IEEE, 2019. DOI: 10.1109/icst46873.2019.9047730. [148] F. Wirth, J. Quehl, J. Ota, and C. Stiller. “PointAtMe: Efficient 3D Point Cloud Labeling in Virtual Reality”. In: IEEE Intelligent Vehicles Symposium. IEEE, 2019. DOI: 10. 1109/ivs.2019.8814115. [149] S. Maneewongvatana and D. M. Mount. “Analysis of Approximate Nearest Neighbor Searching with Clustered Point Sets”. In: ArXiv (1999). DOI: 10.48550/ARXIV.CS/ 9901013. [150] J. Wilhelms and A. V. Gelder. “Octrees For Faster Isosurface Generation”. In: Proceedings of the 1990 Workshop on Volume Visualization - VVS ’90. ACM Press, 1990. DOI: 10.1145/99307.99321. [151] R. B. Rusu, Z. C. Marton, N. Blodow, A. Holzbach, and M. Beetz. “Model-Based and Learned Semantic Object Labeling in 3D Point Cloud Maps of Kitchen Environments”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2009. DOI: 10.1109/iros.2009.5354759. [152] H. Balta, J. Velagic, W. Bosschaerts, G. D. Cubber, and B. Siciliano. “Fast Statistical Outlier Removal Based Method for Large 3D Point Clouds of Outdoor Environments”. In: IFAC-PapersOnLine 51.22 (2018), pp. 348–353. DOI: 10.1016/j.ifacol.2018.11. 566. [153] J. Zhu, J. Gehrung, R. Huang, B. Borgmann, Z. Sun, L. Hoegner, M. Hebel, Y. Xu, and U. Stilla. “TUM-MLS-2016: An Annotated Mobile LiDAR Dataset of the TUM City Campus for Semantic Point Cloud Interpretation in Urban Areas”. In: Remote Sensing 12.11 (2020), p. 1875. DOI: 10.3390/rs12111875. XIII Bibliography [154] CloudCompare. 3D Point Cloud and Mesh Processing Software Open-Source Project. Version 2.12. 2021. URL: http : / / www . cloudcompare . org/ (visited on 06/24/2021). [155] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. “Dynamic Graph CNN for Learning on Point Clouds”. In: ACM Transactions on Graphics 38.5 (2019), pp. 1–12. DOI: 10.1145/3326362. [156] K. Wada. Labelme: Image Polygonal Annotation with Python. 2016. DOI: 10.5281/ zenodo.5711226. URL: https://github.com/wkentaro/labelme (visited on 07/15/2021). [157] E. Gaur, V. Saxena, and S. K. Singh. “Video Annotation Tools: A Review”. In: IEEE International Conference on Advances in Computing, Communication Control and Networking. IEEE, 2018. DOI: 10.1109/icacccn.2018.8748669. [158] X. Roynard, J.-E. Deschaud, and F. Goulette. “Paris-Lille-3D: A Large and HighQuality Ground-Truth Urban Point Cloud Dataset for Automatic Segmentation and Classification”. In: The International Journal of Robotics Research 37.6 (2018), pp. 545–557. DOI: 10.1177/0278364918767506. [159] W. Tan, N. Qin, L. Ma, Y. Li, J. Du, G. Cai, K. Yang, and J. Li. “Toronto-3D: A Largescale Mobile LiDAR Dataset for Semantic Segmentation of Urban Roadways”. In: Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2020. DOI: 10.1109/cvprw50498.2020.00109. [160] C. Wang, Y. Dai, N. Elsheimy, C. Wen, G. Retscher, Z. Kang, and A. Lingua. “ISPRS Benchmark on multisensory indoor mapping and positioning”. In: ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences V-5-2020 (2020), pp. 117–123. DOI: 10.5194/isprs-annals-v-5-2020-117-2020. [161] G. Poier, M. Seidl, M. Zeppelzauer, C. Reinbacher, M. Schaich, G. Bellandi, A. Marretta, and H. Bischof. “PetroSurf3D – A Dataset for high-resolution 3D Surface Segmentation”. In: Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing. ACM, 2017. DOI: 10.1145/3095713.3095719. [162] W. Liu, J. Sun, W. Li, T. Hu, and P. Wang. “Deep Learning on Point Clouds and Its Application: A Survey”. In: Sensors 19.19 (2019), p. 4188. DOI: 10.3390/s19194188. [163] B. Gao, Y. Pan, C. Li, S. Geng, and H. Zhao. “Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey of Datasets and Methods”. In: IEEE Transactions on Intelligent Transportation Systems 23.7 (2020), pp. 6063–6081. ISSN: 1524-9050. DOI: 10.1109/tits.2021.3076844. [164] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Tech. rep. Stanford University, Princeton University, Toyota Technological Institute at Chicago, 2015. arXiv: 1512.03012. XIV Bibliography [165] J. Iqbal, R. Xu, S. Sun, and C. Li. “Simulation of an Autonomous Mobile Robot for LiDAR-Based In-Field Phenotyping and Navigation”. In: Robotics 9.2 (2020), p. 46. DOI: 10.3390/robotics9020046. [166] L. Winiwarter, A. M. E. Pena, H. Weiser, K. Anders, J. M. Sánchez, M. Searle, and B. Höfle. “Virtual Laser Scanning with HELIOS++: A Novel Take on Ray Tracing-Based Simulation of Topographic Full-Waveform 3D Laser Scanning”. In: Remote Sensing of Environment 269 (2021), p. 112772. DOI: 10.1016/j.rse.2021.112772. [167] V. Stojanovic, M. Trapp, J. Döllner, and R. Richter. “Classification of Indoor Point Clouds Using Multiviews”. In: The 24th International Conference on 3D Web Technology. ACM, 2019. DOI: 10.1145/3329714.3338129. [168] J. Gehrung, M. Hebel, M. Arens, and U. Stilla. “An Approach to Extract Moving Objects from MLS Data Using a Volumetric Background Representation”. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences IV-1/W1 (2017), pp. 107–114. DOI: 10.5194/isprs-annals-iv-1-w1-107-2017. [169] E. Grilli, F. Menna, and F. Remondino. “A Review of Point Clouds Segmentation and Classification Algorithms”. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-2/W3 (2017), pp. 339–344. DOI: 10.5194/isprs-archives-xlii-2-w3-339-2017. [170] P. Klinger. “Übersicht und Bewertung von aktuellen Segmentierungsund Klassifizierungsmethoden von Punktwolken”. Bachlorthesis. HafenCity University Hamburg, 2019. [171] P. Babahajiani, L. Fan, J.-K. Kämäräinen, and M. Gabbouj. “Comprehensive Automated 3D Urban Environment Modelling Using Terrestrial Laser Scanning Point Cloud”. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2016. DOI: 10.1109/cvprw.2016.87. [172] O. Hassaan, A. Shamail, Z. Butt, and M. Taj. “Point Cloud Segmentation using Hierarchical Tree for Architectural Models”. In: IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2019, pp. 1582–1586. DOI: 10.1109/icassp. 2019.8682708. [173] E. Castillo, J. Liang, and H. Zhao. “Point Cloud Segmentation and Denoising via Constrained Nonlinear Least Squares Normal Estimates”. In: Mathematics and Visualization. Springer Berlin Heidelberg, 2012, pp. 283–299. DOI: 10.1007/978-3-642-341410_13. [174] P. J. Besl and R. C. Jain. “Segmentation through Variable-Order Surface Fitting”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 10.2 (1988), pp. 167–192. DOI: 10.1109/34.3881. [175] F. Pauling, M. Bosse, and R. Zlot. “Automatic Segmentation of 3D Laser Point Clouds by Ellipsoidal Region Growing”. In: Australasian Conference on Robotics and Automation. Sydney, Australia, 2009. XV Bibliography [176] J. Strom, A. Richardson, and E. Olson. “Graph-Based Segmentation for Colored 3D Laser Point Clouds”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. 2131-2136. IEEE, 2010. DOI: 10.1109/iros.2010.5650459. [177] A. Golovinskiy and T. Funkhouser. “Min-Cut Based Segmentation of Point Clouds”. In: 12th International Conference on Computer Vision Workshops. IEEE, 2009, pp. 39– 46. DOI: 10.1109/iccvw.2009.5457721. [178] F. Moosmann, O. Pink, and C. Stiller. “Segmentation of 3D Lidar Data in non-flat Urban Environments using a Local Convexity Criterion”. In: IEEE Intelligent Vehicles Symposium. IEEE, 2009, pp. 215–220. DOI: 10.1109/ivs.2009.5164280. [179] A. Aijazi, P. Checchin, and L. Trassoudaine. “Segmentation Based Classification of 3D Urban Point Clouds: A Super-Voxel Based Approach with Evaluation”. In: Remote Sensing 5.4 (2013), pp. 1624–1650. DOI: 10.3390/rs5041624. [180] P. V. C. Hough. “Method and Means for Recognizing Complex Patterns”. US Patent 3069654. 1962. [181] D. Borrmann, J. Elseberg, K. Lingemann, and A. Nüchter. “The 3d Hough Transform for Plane Detection in Point Clouds: A Review and a New Accumulator Design”. In: 3D Research 2.2 (2011). ISSN: 2092-6731. DOI: 10.1007/3dres.02(2011)3. [182] W. Shi, W. Ahmed, N. Li, W. Fan, H. Xiang, and M. Wang. “Semantic Geometric Modelling of Unstructured Indoor Point Cloud”. In: ISPRS International Journal of Geo-Information 8.1 (2018), p. 9. DOI: 10.3390/ijgi8010009. [183] M. Weinmann, B. Jutzi, and C. Mallet. “Feature Relevance Assessment for the Semantic Interpretation of 3D Point Cloud Data”. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences II-5/W2 (2013), pp. 313– 318. DOI: 10.5194/isprsannals-ii-5-w2-313-2013. [184] M. Weinmann, B. Jutzi, and C. Mallet. “Semantic 3D Scene Interpretation: A Framework Combining Optimal Neighborhood Size Selection with Relevant Features”. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences II-3 (2014), pp. 181–188. DOI: 10.5194/isprsannals-ii-3-181-2014. [185] M. Weinmann, A. Schmidt, C. Mallet, S. Hinz, F. Rottensteiner, and B. Jutzi. “Contextual Classification of Point Cloud Data by Exploiting Individual 3d Neigbourhoods”. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences II-3/W4 (2015), pp. 271–278. DOI: 10.5194/isprsannals-ii-3-w4-271-2015. [186] H. Thomas, F. Goulette, J.-E. Deschaud, B. Marcotegui, and Y. LeGall. “Semantic Classification of 3D Point Clouds with Multiscale Spherical Neighborhoods”. In: IEEE International Conference on 3D Vision. IEEE, 2018. DOI: 10.1109/3dv.2018.00052. [187] Z. Lari and A. Habib. “Alternative Methodologies for the Estimation of Local Point Density Index: Moving Towards Adaptive Lidar Data Processing”. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XXXIX-B3 (2012), pp. 127–132. DOI: 10.5194/isprsarchives-xxxix-b3-127-2012. XVI Bibliography [188] F. Poux and R. Billen. “Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs. Deep Learning Methods”. In: ISPRS International Journal of Geo-Information 8.5 (2019), p. 213. DOI: 10.3390/ ijgi8050213. [189] F. Poux, C. Mattes, and L. Kobbelt. “Unsupervised Segmentation of Indoor 3D Point Cloud: Application to Object-Based Classification”. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIV-4/W12020 (2020), pp. 111–118. DOI: 10.5194/isprs-archives-xliv-4-w1-2020-111-2020. [190] E. Grilli, D. Dininno, G. Petrucci, and F. Remondino. “From 2D to 3D Supervised Segmentation and Classification for Cultural Heritage Applications”. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-2 (2018), pp. 399–406. DOI: 10.5194/isprs-archives-xlii-2-399-2018. [191] S. Teruggi, E. Grilli, M. Russo, F. Fassi, and F. Remondino. “A Hierarchical Machine Learning Approach for Multi-Level and Multi-Resolution 3D Point Cloud Classification”. In: Remote Sensing 12.16 (2020), p. 2598. DOI: 10.3390/rs12162598. [192] E. Grilli, F. Poux, and F. Remondino. “Unsupervised Object-Based Clustering in Support of Supervised Point-Based 3d Point Cloud Classification”. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2021 (2021), pp. 471–478. DOI: 10.5194/isprs-archives-xliii-b2-2021-4712021. [193] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin. “Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation”. In: Conference on Computer Vision and Pattern Recognition. IEEE, 2020. DOI: 10.1109/cvpr46437. 2021.00981. [194] H. Radi and W. Ali. “VolMap: A Real-time Model for Semantic Segmentation of a LiDAR surrounding view”. In: ICML (Thirty-sixth International Conference on Machine Learning) Workshop on AI for Autonomous Driving. arXiv, 2019. [195] R. A. Rosu, P. Schütt, J. Quenzel, and S. Behnke. “LatticeNet: Fast Point Cloud Segmentation Using Permutohedral Lattices”. In: Proceedings of Robotics: Science and Systems. Robotics: Science and Systems Foundation, 2020. DOI: 10.15607/rss. 2020.xvi.006. [196] H.-Y. Meng, L. Gao, Y.-K. Lai, and D. Manocha. “VV-Net: Voxel VAE Net With Group Convolutions for Point Cloud Segmentation”. In: International Conference on Computer Vision. IEEE, 2019. DOI: 10.1109/iccv.2019.00859. [197] A. Krizhevsky. “One weird trick for parallelizing convolutional neural networks”. In: arXiv (2014). DOI: 10.48550/ARXIV.1404.5997. arXiv: 1404.5997. [198] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. “You Only Look Once: Unified, Real-Time Object Detection”. In: IEEE Conference on Computer Vision and Pattern Recognition. 2015, pp. 779–788. DOI: 10.1109/cvpr.2016.91. XVII Bibliography [265] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu. “Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes”. In: Conference on Computer Vision and Pattern Recognition. IEEE, 2019, pp. 8868–8876. DOI: 10.1109/cvpr.2019. 00908. [266] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. “Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition”. In: Neural networks. Vol. 32. 32. Elsevier BV, 2012, pp. 323–332. DOI: 10.1016/j.neunet.2012.02.016. [267] G. Joos. “Zur Qualität von objektstrukturierten Geodaten”. PhD thesis. Universität der Bundeswehr München, 2000. [268] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. “3D Semantic Parsing of Large-Scale Indoor Spaces”. In: Conference on Computer Vision and Pattern Recognition. IEEE, 2016, pp. 1534–1543. DOI: 10.1109/cvpr.2016.170. [269] M. A. Uy, Q.-H. Pham, B.-S. Hua, D. T. Nguyen, and S.-K. Yeung. “Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on RealWorld Data”. In: International Conference on Computer Vision. IEEE, 2019. DOI: 10. 1109/iccv.2019.00167. [270] T. Yu and H. Zhu. “Hyper-Parameter Optimization: A Review of Algorithms and Applications”. In: arXiv arXiv:2003.05689 (2020). arXiv: 2003.05689v1 [cs.LG]. [271] D. Griffiths and J. Boehm. “Weighted Point Cloud Augmentation for Neural Network Training Data Class-Imbalance”. In: ISPRS The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-2/W13 (2019), pp. 981–987. DOI: 10.5194/isprs-archives-XLII-2-W13-981-2019. [272] J. Morel, A. Bac, and T. Kanai. “Segmentation of Unbalanced and In-Homogeneous Point Clouds and Its Application to 3D Scanned Trees”. In: The Visual Computer 36.10-12 (2020), pp. 2419–2431. DOI: 10.1007/s00371-020-01966-7. [273] V. Kasireddy and B. Akinci. “Assessing the Impact of 3D Point Neighborhood Size Selection on Unsupervised Spall Classification with 3D Bridge Point Clouds”. In: Advanced Engineering Informatics 52 (2022), p. 101624. DOI: 10.1016/j.aei.2022. 101624. [274] Y. Liu, B. Fan, G. Meng, J. Lu, S. Xiang, and C. Pan. “DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing”. In: International Conference on Computer Vision. IEEE, 2019. DOI: 10.1109/iccv.2019.00534. [275] B.-S. Hua, Q.-H. Pham, D. T. Nguyen, M.-K. Tran, L.-F. Yu, and S.-K. Yeung. “SceneNN: A Scene Meshes Dataset with aNNotations”. In: Fourth International Conference on 3D Vision. IEEE, 2016, pp. 92–101. DOI: 10.1109/3dv.2016.18. [276] H. Zhao, L. Jiang, C.-W. Fu, and J. Jia. “PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing”. In: Conference on Computer Vision and Pattern Recognition. IEEE, 2019. DOI: 10.1109/cvpr.2019.00571. XXIV Bibliography [277] S. Ando and C. Y. Huang. “Deep Over-sampling Framework for Classifying Imbalanced Data”. In: European Conference, ECML PKDD. Springer International Publishing, Cham, 2017, pp. 770–785. DOI: 10.1007/978-3-319-71249-9_46. [278] M. Joseph-Rivlin, A. Zvirin, and R. Kimmel. “Momenet: Flavor the Moments in Learning to Classify Shapes”. In: International Conference on Computer Vision Workshop. IEEE, 2019. DOI: 10.1109/iccvw.2019.00503. XXV A Peer-reviewed publications A.1 Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet Reference: Barnefske, E. & Sternberg, H. (2020): Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet. In Wunderlich, T. (Ed.), Ingenieurvermessung 20. Beiträge zum 19. Internationalen Ingenieurvermessungskurs München, 2020, Herbert Wichmann Verlag, 2020, 127-139. Contribution of Co-Authors: Table 6: Contribution to Paper No. 1 Involved in Estimated contribution Ideas and conceptual design 90% Computation and results 100% Analysis and interpretation 95% Manuscript, figures and tables 100% Total: 96% I hereby confirm the correctness of the declaration of the contribution of Eike Barnefske for Paper No. 1 in Table 6: Prof. Dr.-Ing. Harald Sternberg, HafenCity Universität Hamburg XXVII A Peer-reviewed publications A.1 Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet Reference: Barnefske, E. & Sternberg, H. (2020): Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet. In Wunderlich, T. (Ed.), Ingenieurvermessung 20. Beiträge zum 19. Internationalen Ingenieurvermessungskurs München, 2020, Herbert Wichmann Verlag, 2020, 127-139. Contribution of Co-Authors: Table 6: Contribution to Paper No. 1 Involved in Estimated contribution Ideas and conceptual design 90% Computation and results 100% Analysis and interpretation 95% Manuscript, figures and tables 100% Total: 96% I hereby confirm the correctness of the declaration of the contribution of Eike Barnefske for Paper No. 1 in Table 6: Prof. Dr .-Ing. Harald Sternberg, HafenCity Universität Hamburg XXVII 1 T. Wunderlich (Hrsg.), Ingenieurvermessung 2020 © Herbert Wichmann Verlag Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet Eike BARNEFSKE und Harald STERNBERG 1 Einleitung Punktwolken dienen als einfaches Modell oder als Grundlage von Planungen, geometrischen Analysen und komplexen Modellierungen. Häufig sind im ersten Schritt Teilpunktwolken für diese Aufgaben zu erstellen. Eine Filterung nach fehlerhaft gemessenen Punkten oder ein modellbasiertes Reduzieren der Punktwolkendichte sind häufig die ersten Klassifizierungen, die auf eine Punktwolke angewendet werden. Hierfür werden vorrangig händische oder modellbasierte Verfahren genutzt. Alternativ zu modellbzw. wissensbasierten Klassifizierungsverfahren werden datenbasierte Klassifizierungen entwickelt, um die Auswertungszeit zu reduzieren, die Klassifizierungsqualität bei unterschiedlichen Aufnahmesystemen und Szenen zu steigern, sowie um die Klassifizierung zu automatisieren. Hierfür finden u. a. Convolutional Neurale Netzwerke (ConvNet) Anwendung, die das Wissen aus den klassifizierten Punktwolken (Trainingsdaten) lernen und dieses in der Anwendungsphase nutzen, um unbekannte Punkte zu klassifizieren. Die Klassifizierungsleistung der ConvNet wird folglich durch die Netzwerkarchitektur und die Trainingsdaten (Wissen des Algorithmus) beeinflusst. Arbeiten von QI ET AL. (2017A), HACKEL ET AL. (2017) u. a. zur semantischen Klassifizierung mit ConvNet richten sich an folgendes Ziel: Es sollen die Punkte identifiziert werden, welche bestimmte Objekte in der Aufnahmeszene (z. B. Bäume oder Straßen) beschreiben. Hierbei werden aber fehlerhaft gemessene Punkte, die i. d. R. nur wenige Prozent der gesamten Punktwolke einnehmen, nicht betrachtet und einfach einer Objektklasse zugeordnet. Diese fehlerhaften Punkte treten z. B. in der Form von Mixed Pixel, Mehrwegeeffekten, einem großen Oberflächenrauschen oder Phantompunkten auf. Der Einfluss der fehlerhaft gemessenen Punkte auf die semantische Klassifizierung und eine Klassifizierung nach fehlerhaften gemessenen Punkten ist Gegenstand der Untersuchungen dieser Arbeit. Hierfür werden händisch klassifizierte Punktwolken in unterschiedlichen Klassenkombinationen für das Training und die Evaluierung der Klassifizierungsleistung verwendet. Es werden die Unterschiede und Ähnlichkeiten von modellund datenbasierten Klassifikationsverfahren vorgestellt (Abschnitt 2.1). Die Funktionsweise von ConvNet (Abschnitt 2.2) und verschiedene ConvNet-Modelle für strukturierte und unstrukturierte Punktwolken werden erläutert (Abschnitte 2.3 und 2.4). Der Einfluss bei vier unterschiedlichen Klassenkombinationen mit und ohne fehlerhaften Punkten auf die Klassifizierungsleistung wird am Beispiel der Netzwerkarchitektur von PointNet untersucht (Abschnitte 3.1 bis 3.3). Aufbauend auf den Beobachtungen der Untersuchungen werden Strategien für das Training von ConvNet zur Klassifizierung von fehlerhaft gemessenen Punkten vorgestellt. Zudem werden Ideen vorgestellt, um den Einfluss von fehlerhaft gemessenen Punkten bei Klassifizierungen zu minieren (Abschnitt 3.4). XXVIII 2 Eike Barnefske und Harald Sternberg 2 Klassifizierung von Punktwolken Punktwolken stellen geometrisch einen erfassten Raum dar, lassen aber ohne weiteres Wissen eine semantische Trennung von einzelnen Objekten in der erfassten Szene nicht zu. Diese semantische Zerlegung der Punktwolke in Unterpunktwolken ist ein wichtiger Schritt des Auswertungsprozesses von Punktwolken, damit zum einen fehlerhaft gemessene Punkte aus der Punktwolke entfernt werden und zum anderen Objekte in einer Szene semantisch unterschieden werden können. Eine semantische Unterscheidung ist wichtig, da nicht alle erfassten Objekte für eine Fragestellung von Interesse sind. Für die Erstellung von Stadtmodellen sind z. B. Bauwerke von Interesse, Fahrzeuge hingegen werden hier als störende Objekte deklariert. Soll die Punktwolke für die Analyse des Verkehrsraums, z. B. bei der Verkehrsplanung oder der Navigation, verwendet werden, sind Punkte, die Objekte auf den Verkehrswegen (Autos, Fußgänger und Radfahrer) erfassen, von vornehmlichem Interesse. Die Trennung von Punkten, die Objekte beschreiben und Punkten, die aufgrund von Messfehlern entstanden sind, wird i. d. R mit Filtern durchgeführt. Diese Filter nutzen allgemeine oder sensorspezifische Modelle zur Unterscheidung, ob ein Punkt zu einem Objekt gehört oder aufgrund eines Messfehlers entstanden ist. Zudem werden die Filter zum Homogenisieren der Punktwolkendichte und zur Auswahl von Punktwolkenabschnitten eingesetzt. Filter haben den Nachteil, dass viel Wissen über die Punktwolke vorhanden sein muss und dieses für jeden möglichen Fall angewendet werden muss. In einem nachfolgenden Schritt kommen i. d. R. andere Modelle zur automatischen semantischen Trennung der Punktwolken nach Objektklassen zum Einsatz. Beispielhaft werden diese modellbasierten Verfahren vorgestellt. Im Gegensatz zu den modellbasierten Klassifizierungsverfahren werden vermehrt datenbasierte Klassierungsverfahren entwickelt, die die Trennung der Punktwolken nicht aufgrund von vorgegebenem Wissen, sondern von erlerntem Wissen für die semantische Trennung der Punktwolken durchführen. Den populärsten Ansatz stellen zurzeit ConvNet dar, dessen Einsatz an einigen Beispielen vorgestellt wird. 2.1 Modellbasierte Segmentierung und Klassifizierung Die Klassifizierung von Punktwolken basiert auf den vier zentralen Arbeitsschritten, und zwar (1) der Detektion von Merkmalen, (2) dem Sortieren der Punkte nach diesen Merkmalen, (3) dem Festlegen von Grenzen, die die Gruppen (Segmente) mit ähnlichen Merkmalen voneinander unterscheiden, und (4) dem Zuweisen eines Klassennamens an alle Segmente mit gleichen bzw. ähnlichen Merkmalen. Die Arbeitsschritte 1 bis 3 werden als Segmentierung bezeichnet, auf die eine Klassifikation folgen kann. Bei modellbasierten Verfahren können diese Arbeitsschritte i. d. R. eindeutig von dem der Klassifikation unterschieden werden. Dies ist bei datenbasierten Verfahren zunehmend nicht mehr möglich, da die Generierung der Segmente aufgrund von Klassenmerkmalen in einem Schritt erfolgt. Eine modellbasierte Auswertung von Punktwolken hingegen ist in verschiedenen Auswertestufen gut zu unterteilen. In jeder Stufe wird nach einem bestimmten und beschreibbaren Merkmal in der Punktwolke gesucht. Punkte, die das gesuchte Merkmal mit ähnlicher Ausprägung tragen, werden als Segmente oder Klassen zusammengefasst. Ein Beispiel hierfür ist die Unterteilung von Punktwolken in zwei oder mehrere Segmente in Abhängigkeit von der Distanz zum Aufnahmestandort (Merkmal ist hier die Distanz). Die erste Auswertungsstufe ist häufig das „Filtern“ von fehlerhaften Punkten, deren Auftreten in der Punktwolke zum Teil beschrieben werden kann. Dieser Stufe folgen verschiedene weitere Stufen, in denen Segmentierungund XXIX Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet 3 Klassifikationsverfahren mit dem Ziel der Generierung von Objektklassen angewendet werden. Die graphbasierte Segmentierung von Punktwolken ist ein weit verbreitetes Verfahren, anhand dessen die Arbeitsschritte 1 bis 3 der Segmentierung gut nachzuvollziehen sind. Die einzelnen Punkte der Punktwolke stellen die Knoten des Graphen dar, die durch Kanten miteinander verbunden sind. Jede Kante erhält, aufgrund der Merkmalsunterschiede zwischen den Punkten, ein oder mehrere Gewichte, die gemessen oder berechnet werden (1). STORM ET AL., (2010) nutzen z. B. Farbinformationen (RGB-Werte), euklidische Distanzen und die Richtung der Punktnormalen, die über ein lokales Netz berechnet werden. Weitere Merkmale, die Laserscanner messen und für die Unterscheidung von Objekten einen Mehrwert darstellen, sind die Intensität oder die Rückkehrreihenfolge des empfangenen Signals. Bei der graphbasierten Segmentierung werden die Kanten mit den dazugehörigen Punkten anhand der Gewichte eines Merkmales, i. d. R. absteigend, sortiert (2) und ein Startgrenzwert für jedes Gewicht wird festgelegt (3). In einem iterativen Prozess werden Punkte einem Segment zugeordnet, Segmente zusammengefasst oder neue Segmente erstellt. Die Entscheidung, ob Segmente zusammengefasst oder neue Segmente gebildet werden, wird durch einen Grenzwert oder durch alle Grenzwerte bestimmt (FELZENSZWALB & HUTTENLOCHER, 2004). Erweiterungen des Algorithmus sehen ein dynamisches Anpassen der Grenzwerte vor, um optimale und detaillierte Segmente zu berechnen. Dieses Verfahren wird häufig um VoxelGitter erweitert (wie bei AIJAZI ET AL., 2013), da bei unstrukturierten Punktwolken durch ein festes oder ein dynamisches Gitter die Auswertung vereinfacht und beschleunigt werden kann. Die graphbasierte Segmentierung kann auf verschiedenen Oberflächen, wie vermaschten Punktwolken, Voxel-Gittern oder Oberflächenmodellen, erfolgen. Bei der Erstellung dieser Modelle erfolgt immer eine Generalisierung der Messwerte, so dass eine punktscharfe Segmentierung, wie sie für die Klassifikation von Messfehlern notwendig wäre, nicht mehr möglich ist. 2.2 Klassifizierung mit ConvNet ConvNets werden für die detaillierte und automatische Klassifikation von Bildern eingesetzt, um die Inhalte der Bilder automatisch zu entschlüsseln und diese nach semantischen Aspekten zu clustern (GIRSHICK ET AL., 2014, GIRSHICK, 2015, REN ET AL., 2016). In digitalen Bildern sind die Merkmale, die für die Klassifikation von Objekten verwendet werden, in gleichmäßigen und gleich großen Rastern (Pixel) angeordnet. Diese Anordnung der Merkmale und die scharfe Abgrenzung der Merkmale bei gleichzeitiger lückenloser Verfügbarkeit ermöglichen ein sofortiges und effizientes Verarbeiten der Bilder mit Verfahren der Matrizenrechnungen. Mittels der Merkmale in den Eingabebildern und dessen Nachbarschaft, werden neue multidimensionale Merkmale in einer Convolutional-Schicht extrahiert. Neue Merkmale werden durch das Multiplizieren der Information mit festen Gewichten, die in einem einoder mehrdimensionalen Filter angeordnet sind, bestimmt. Die Größe des Filters und die Gewichte, werden für die Klassifizierungsaufgabe so ausgewählt, dass eindeutige Merkmale bestimmt werden können (Abb. 1). XXX 4 Eike Barnefske und Harald Sternberg Abb. 1: Funktion einer Convolutional-Schicht am Beispiel einer 5 x 5 Eingabe und eines 3 x 3 Filters ohne Ausfüllen des Filters. Im Anschluss an diese Schicht können weitere Convolutionaloder PoolingSchichten folgen. Liegt für jeden Merkmalsträger mehr als ein Merkmal vor (dieses ist z. B. der Fall, wenn ein Bild aus drei Farbkanälen besteht), dann wird der Filter auf jeden Informationskanal angewendet und die Summe der neuen Merkmale je Träger bestimmt. Die Merkmale aus einer oder mehreren Convolution-Schichten werden durch das Pooling aggregiert. Hierbei wird ein weiteres Raster über eine feste Anzahl an Merkmalsträgern gelegt und die Merkmale werden zu einem Wert in einer Rasterzelle zusammengefasst. Hierfür wird meist der größte, der kleinste oder der mittlere Merkmalswert verwendet. Die eigentliche Klassifizierung wird durch ein „normales“, künstliches neurales Netz (KNN) durchgeführt, in dem alle Merkmale der letzten Convolution-Pooling-Schicht mit den möglichen Netzausgaben (Klassen) verknüpft werden, so dass ein Vektor aufgestellt wird, der für jede Klasse eine Netzausgabe ausgibt. Funktionen wie Softmax, die auf dem Vektor angewendet werden, ermöglichen das Bestimmen der wahrscheinlichsten Klasse für jeden Merkmalsträger bzw. jedes Pixel (SZE ET AL., 2017). 2.3 ConvNet für Punktwolkenklassifikationen mit Gitterstrukturen Punktwolken sind i. d. R. unsortiert, weisen regional unterschiedliche Punktdichten und eine unregelmäßige Verteilung der Merkmale auf, so dass viele Punktwolkenklassifizierungsverfahren einen Zwischenschritt benötigen. Dieser Zwischenschritt hat das Ziel, die Punktwolken in eine Struktur zu überführen, die Pixel oder Voxel nutzt. Hierfür werden die Punktwolken in andere Räume projiziert und Informationen zusammengefasst. Details gehen durch diese Vorverarbeitung verloren und fehlerhafte Punkte, die nur vereinzelt auftreten, werden bei der späteren Klassifizierung fälschlicherweise einer anderen Objektgruppe zugeordnet. Anwendungen, bei denen größere einzelne Objekte in der Punktwolke während der Aufnahme zu klassifizieren bzw. durch eine Bounding Box zu markieren sind, sind aktuell nur durch eine Generalisierung der Punktwolke vor der Klassifizierung möglich. PIXOR (YANG ET AL., 2018) ist ein ConvNet für die Klassifizierung von Fahrzeugen und deren Bewegungsrichtung in dreidimensionalen Punktwolken. Hierfür wird eine Generalisierung durch das Erzeugen einer Vogelperspektivenansicht durchgeführt. Diese 2D-Ansicht wird in Voxel / Pixel unterteilt. Auf dieser vorverarbeiteten Punktwolke können die 2D-ConvNet angewendet werden. XXXI Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet 5 Die Voxel-Struktur wird in einer Vielzahl von Arbeiten als Grundlage für ein occupancy grid verwendet. Ein occupancy grid ist eine Rasterstruktur, in der die Zellen dem Zustand „vorhanden sein“ oder „nicht vorhandenen sein“ von Punkten zugeordnet werden. Durch dieses Raster werden die Punktwolken abstrahiert. Ziel der Verwendung des occupancy grids ist es, eine Punktwolke effizient in die 26 Klassen des Sydney Urban Objects Dataset (DEUGE ET. AL., 2013) zu unterteilen. Bei VoxNet (MATURANA & SCHERER, 2015) wird die Punktwolke in quadratische Voxel-Segmente unterteilt. Jedes Voxel-Segment wird wiederum in 32³ Subvoxel unterteilt, für die ein Wert für den Besetzungszustand (z. B. binärer oder als punktdichte Wert) berechnet wird. Jedes Voxel-Segment ist ein Tensor aus 32 x 32 x 32 Einträgen. Dieser Tensor wird an ein 3D-ConvNet übergeben und für das Voxel-Segment wird eine Klasse bestimmt, der alle Punkte, die in dieses Segment fallen, zugeordnet werden. HACKEL ET AL. (2017) folgen diesem Verfahren, berechnen aber um jeden Punkt der Punktwolke ein 16 x 16 x 16 großes Voxel-Gitter bei fünf unterschiedlich großen Kantenlängen (von 2,5 bis 40 cm). Für jeden der Voxel wird ein Besetzungszustand berechnet, so dass ein 5 x 16 x 16 x 16 Tensor entsteht, der die geometrische Nachbarschaft des Punktes beschreibt. Die Merkmale dieses Tensors für jeden Punkt werden mit einem ConvNet in Anlehnung an das ConvNet VGG von SIMONYAN & ZISSERMAN (2014) verarbeitet, so dass für jeden Punkt die Klassifizierung durch den Softmax-Layer (Klassifizierungsfunktion) erfolgt. Dieser Ansatz, der punktorientierten Klassifizierung von komplexen dreidimensionalen Punktwolken, wird bei PointNet und dessen Erweiterungen weiterverfolgt. 2.4 PointNet und Erweiterungen Das PointNet (QI ET AL. 2017A) in seiner Grundform besteht aus einer Eingabeschicht, in der eine vollständige, kleine Punktwolke (2000 bis 4000 Punkte) oder ein Punktwolkensegment (Ausschnitt einer großen Punktwolke) als Tensor verarbeitet wird. Die Punktwolke bzw. der Tensor besteht mindestens aus den Punkten (n) mit Koordinatentripel (obligatorisch) und den optionalen Merkmalen (m), wie Normalen-Vektoren der Punkte, RGBoder Intensitätswerten. Die Werte des eingelesenen Tensors werden durch ein T-Net, eine ConvNet für eine Starrkörpertransformation, in den Schwerpunkt des Punktwolkensegments transformiert. Diese Transformation kann sowohl auf Merkmale als auch auf Koordinaten angewendet werden. Nach der Transformation sind die Verarbeitungsschritte des PointNet, die hochdimensionale Merkmalsextraktion, die Sortierung und das Zusammenfassen von Merkmalen, so dass Punkte aufgrund der Merkmale einer Klasse zugeordnet werden können. Die Funktionsweise von PointNet entspricht dabei zweier verketteter Funktionen. Die innere Funktion extrahiert die Merkmale auf Grundlage der Merkmale der vorherigen Schichten. Dieses wird durch Multilayer Perceptron (MLP) erreicht. MLP sind mehr Schichten von verketten Noten eines KNN. Die äußere Funktion ist die sortierende bzw. aggregierende Funktion, die Merkmale zusammenfasst. Diese wird durch eine Max-Pooling–Schicht umgesetzt. Für die Segmentierung bzw. punktweise Klassifikation werden lokale und globale Merkmale miteinander kombiniert. D. h. ein neuer Tensor mit den Dimensionen (n x mlokal + mgobal), der aus den aggregierten Merkmalen und den lokalen Merkmalen jedes Punktes besteht, wird erstellt. Aus diesem Tensor werden wieder neue Merkmale je Punkt extrahiert und aggregiert. Für jeden Punkt werden Merkmale durch die MLP zusammengefasst, so dass eine Klassifizierung, in k vorgegebenen Klassen, erfolgen kann. Diese Klassifizierung erfolgt aufgrund des höchsten Wertes des Klassenvektors jedes Punktes (Abb. 2). XXXII 6 Eike Barnefske und Harald Sternberg Abb. 2: Vereinfachte Darstellung des PointNet Verfahrens zur Segmentierung und Klassifizierung von Punktwolken in Anlehnung an QI ET AL., (2017A). Multilayer Perceptron (MLP) werden für die Extraktion von Merkmalen verwendet, die durch eine Max Pooling Funktion zusammengefasst werden. Die Concat-Funktion kombiniert Tensoren. Für jeden Punkt wird der höchste Ausgabewert aus den vorgegebenen Klassen bestimmt und so klassifiziert. PointNet in dieser Grundform kann nur Merkmale nutzen, die im Punktwolkensegment vorhanden sind. Bei großen und unterschiedlich dichten Punktwolken führt dies zu fehlerhaften Klassifikationsergebnissen. QI ET AL., (2017B) nutzen PointNet als ein Baustein, führen aber eine Struktur von unterschiedlichen Schichten ein, in denen eine große Punktwolke schrittweise verkleinert wird (PointNet++). Aus der Punktwolke werden Punkte mit dem farthestpoint-sample (fps) -Algorithmus ausgewählt, die das Zentrum einer Region bilden. Die Punkte, die zu dieser Region gruppiert werden, werden über einen festen Radius ausgewählt, wodurch die Größe der Region immer konstant ist, aber die Anzahl der Punkte variiert. Für jede Region wird auf Grundlage der Merkmale ein regionaler Merkmalsvektor durch PointNet berechnet, so dass für jede Region ein neuer Merkmalsvektor entsteht. Aus allen neuen Merkmalsvektoren werden in gleicher Weise in der folgenden Schicht neue Merkmalsvektoren berechnet. Wenn ein bestimmtes Abstraktionslevel für die Punktwolke, bzw. nun die Merkmale erreicht ist, dann werden diese Merkmale in der Segmentierungsphase wieder entschlüsselt. Schichtweise werden die Merkmale an die Zentralpunkte übertragen. Die Merkmale werden durch eine Interpolation an die benachbarten Punkte in der jeweiligen Schicht übertragen, so dass alle Punkte einen Merkmalsvektor mit ihren Merkmalen der zwei vorherigen Schichten haben. Mittels eines PointNet Bausteins werden für jeden Punkt aus diesen Merkmalen neue Merkmale aggregiert. In der letzten Segmentierungsschicht hat jeder Punkt ein Set an Merkmalen, welches für die Klassifizierung jedes Punktes verwendet wird. Hierbei liegt die Annahme vor, dass sich Merkmale gleichmäßig ausbreiten, was aber bei Punktwolken mit heterogenen Objektvorkommen nicht zwangsläufig ist. PointNet++ kann um Berechnungsschritte, die die unterschiedliche Punktdichte berücksichtigen, erweitert werden. ENGELMANN ET AL., (2017) adressieren ebenfalls das PointNet Problem, dass keine Merkmale außerhalb eines Punktsegmentes geteilt werden und optimieren die Klassifikationsleistung durch das Teilen von Merkmalen zwischen benachbarten Punktsegmenten. Dieses entspricht der natürlichen Merkmalsausbreitung in Punktwolken mit einer Vielzahl von unterschiedlichen Objekten. In dieser Erweiterung werden zwei Prozessketten vorgestellt, die zum einen die Eingabeschicht und zum anderen die Aggregation von Merkmalen verschiedener XXXIII Klassifizierung von fehlerhaft gemessenen Punkten in 3D-Punktwolken mit ConvNet 13 BARNEFSKE, E. & STERNBERG, H. (2019): PCCT: A Point Cloud Classification Tool To Create 3D Training Data To Adjust And Develop 3D ConvNet. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4216, 35-40. DEUGE, M. D., QUADROS, A., HUNG, C., DOUILLARD, B. (2013): Unsupervised Feature Learning for Classification of Outdoor 3D Scans. Australasian Conference on Robotics and Automation. ENGELMANN, F., KONTOGIANNIA, T., HERMANS, A. & LEIBE, B. (2017): Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 716-124. FELZENSZWALB, P. F. & HUTTENLOCHER, D. P. (2004): Efficient Graph-based Image Segmentation. International Journal of Computer Vision, 59, 2, 167–181. GIRSHICK, R. (2015): Fast R-CNN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1440–1448. GIRSHICK, R., DONAHUE J., DARRELL, T. & MALIK, J. (2014): Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 580–587. GOODFELLOW, I., BENGIO, Y. & COURVILLE, A. (2016): Deep Learning. The MIT Press. HACKEL, T., SAVINOV, N., LADICKY, L., WEGNER, J.D., SCHINDLER, K. & POLLEFEYS, M. (2017): SEMANTIC3D.NET: A New Large-scale Point Cloud Classification Benchmark. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 91–98. MATURANA, D. & SCHERER, S. (2015): VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition. Proceedings of IEEE International Conference on Intelligent Robots and Systems, 922–928. QI, C. R., SU, H., MO, K. & GUIBAS, L. J. (2017A): PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 77-85. QI, C. R.; YI, L.; SU, H. & GUIBAS, L. J. (2017B): PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Advances in Neural Information Processing Systems, 5099-5108 REN, S., HE, K., GIRSHICK, R. & SUN, J (2016): Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems, 91–99. SIMONYAN, K. & ZISSERMAN, A. (2014): Very Deep Convolutional Networks for Largescale Image Recognition, Proceedings of the ICLR, 1409-1556. STROM, J., RICHARDSON, A. & OLSON, E. (2010): Graph-based Segmentation for Colored 3D Laser Point Clouds. Proceedings of IEEE International Conference on Intelligent Robots and Systems, 2131-2136. SZE, V., CHEN, Y.-H., YANG, T.-J. & EMER, J.S. (2017): Efficient Processing of Deep Neural Networks: A tutorial and survey. Proceedings of the IEEE, 105, 12, 2295-2329. YANG, B., LUO, W. & URTASUN, R. (2018): PIXOR: Real-Time 3D Object Detection from Point Clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7652–7660. XL A. Peer-reviewed publications A.2 Evaluating the Quality of Semantic Segmented 3D Point Clouds Reference: Barnefske, E.& Sternberg, H. (2022): Evaluating the Quality of Semantic Segmented 3D Point Clouds. Remote Sensing, 14, 446. DOI: 10.3390/rs14030446 Graphical Abstract: Figure 42: Graphical Abstract: Evaluation of semantic segmentation methods using the quality model. Contribution of Co-Authors: Table 7: Contribution to Paper No. 2 Involved in Estimated contribution Ideas and conceptual design 90% Computation and results 100% Analysis and interpretation 95% Manuscript, figures and tables 100% Total: 96% I hereby confirm the correctness of the declaration of the contribution of Eike Barnefske for Paper 2 in Table 7: Prof. Dr.-Ing. Harald Sternberg, HafenCity Universität Hamburg XLI A. Peer -reviewed publications A. Peer-reviewed publicationsA. Peer A.2 Evaluating the Quality of Semantic Segmented 3D Point Clouds Reference: Barnefske, E.& Sternberg, H. (2022): Evaluating the Quality of Semantic Segmented 3D Point Clouds. Remote Sensing, 14, 446. DOI: 10.3390/rs14030446 Graphical Abstract: Figure 42: Graphical Abstract: Evaluation of semantic segmentation methods using the quality model. Contribution of Co-Authors: Table 7: Contribution to Paper No. 2 Involved in Estimated contribution Ideas and conceptual design 90% Computation and results 100% Analysis and interpretation 95% Manuscript, figures and tables 100% Total: 96% I hereby confirm the correctness of the declaration of the contribution of Eike Barnefske for Paper 2 in Table 7: Prof. Dr .-Ing. Harald Sternberg, HafenCity Universität Hamburg XLI   Citation: Barnefske, E.; Sternberg, H. Evaluating the Quality of Semantic Segmented 3D Point Clouds. Remote Sens. 2022,14, 446. https://doi.org/ 10.3390/rs14030446 Academic Editor: Sander Oude Elberink Received: 20 December 2021 Accepted: 13 January 2022 Published: 18 January 2022 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). remote sensing Article Evaluating the Quality of Semantic Segmented 3D Point Clouds Eike Barnefske * and Harald Sternberg Department of Hydrography and Geodesy, HafenCity University Hamburg, Henning-Voscherau-Platz 1, 20457 Hamburg, Germany; harald.sternber[email protected] *Correspondence: [email protected] Abstract: Recently, 3D point clouds have become a quasi-standard for digitization. Point cloud processing remains a challenge due to the complex and unstructured nature of point clouds. Currently, most automatic point cloud segmentation methods are data-based and gain knowledge from manually segmented ground truth (GT) point clouds. The creation of GT point clouds by capturing data with an optical sensor and then performing a manual or semi-automatic segmentation is a less studied research field. Usually, GT point clouds are semantically segmented only once and considered to be free of semantic errors. In this work, it is shown that this assumption has no overall validity if the reality is to be represented by a semantic point cloud. Our quality model has been developed to describe and evaluate semantic GT point clouds and their manual creation processes. It is applied on our dataset and publicly available point cloud datasets. Furthermore, we believe that this quality model contributes to the objective evaluation and comparability of data-based segmentation algorithms. Keywords: 3D point cloud; quality model; annotation tools; datasets; evaluation metric; evaluation parameter 1. Introduction A major research topic in geodesy is to digitize activities in construction [ 1 – 3 ], in building maintenance [ 4 , 5 ] and in navigation [ 6 , 7 ]. For the digitization of these tasks, digital building parts and furnishing objects must be formed and processed. Digital models of real-world buildings (digital twins) are needed to make complex and large semantic data interpretable for humans and machines [ 8 ]. The creation of digital twins is often based on 3D point clouds, which are efficiently captured with depth imaging cameras or light imaging, detection and ranging (LIDAR) systems. The 3D point cloud without any semantic features can already be considered a model, since humans can use their knowledge to interpret semantic point groups as single objects. These semantic point groups are, e.g., the objects and scanning artifacts, as shown in Figure 1. Figure 1. Examples of objects (chair and table) and scanning artifacts in a point cloud. Common scanning artifacts are: comet tails, mixed pixels on edges (jump edges), multi-path effects and defused reflections. Remote Sens. 2022,14, 446. https://doi.org/10.3390/rs14030446 https://www.mdpi.com/journal/remotesensing XLII Remote Sens. 2022,14, 446 2 of 41 For the digital processing of point clouds, semantic information has to be given to the point cloud to form semantic segments. The initial semantic segmentation is always performed by humans. For this purpose, different tools can be used to form segments as efficiently, reliably, precisely and correctly as possible and to assign the correct semantic label. The efficiency, reliability, precision and correctness of semantic segmentation are characteristics that describe the quality of a semantic point cloud. These characteristics build the quality model, which describes how well the creation of the semantic point cloud works. Evaluation metrics now become parameters of the quality model, which describe the point cloud characteristics. A comparison of different segmentations is possible with the quality parameter. Method comparisons are common in automatic semantic segmentation [ 9 – 12 ], which typically uses machine learning (ML) and artificial intelligence (AI). For method comparisons, point cloud benchmarks are used [ 13 , 14 ]. Semantic point cloud benchmarks are point clouds for which a semantic ground truth (GT) is given. It is assumed that the GT point clouds are free of semantic and geometric errors. However, unfortunately, in most cases, a complete evaluation of the manually or semi-automatically created semantic point cloud benchmarks is not performed. The characteristics of a semantic point cloud that can be evaluated vary strongly among the published point clouds. In some works, the semantic accuracy of a point cloud is evaluated completely [ 13 ] or by spot checks [ 14 , 15 ]. Other works evaluate only the completeness and correctness of a building model [ 16 ]. Even if some characteristics of the point cloud can be evaluated, then a comparison of the evaluation metric is often not possible, since no uniform metrics are defined. For example, intersection over union (IoU), F1-score, overall accuracy, recall, precision and many others are used to validate the accuracy. The variety problem of the evaluation metric for the case of object detection in images is well known and a tool to translate the evaluation metrics for compression was developed [17]. To the best of our knowledge, a holistic quality model in which availability, integrity and accuracy are represented does not exist for semantic point clouds. Such a quality model has the potential to make the investigation of existing and upcoming GT point cloud datasets comparable. Deviation from reality, the availability of information and applicability to a certain purpose can be determined with our quality model for indoor point clouds. Fundamental for the development of the quality model is the definition of the semantic segmentation, as well as its separation into detection and classification (Section 2.1). The capture methods of 3D point clouds for indoor applications (Section 2.2), the existing point cloud datasets (Section 2.3), as well as the tools for manual and semi-automatic semantic segmentations (Section 2.4) determine the characteristics needed in the quality model. The development of the quality model is derived from a process description (Section 3.1), a class definition (Section 3.2) and a data model (Section 3.3). The quality characteristics and parameters are defined and discussed in Section 3.4. The descriptive and evaluative use of the quality model is presented and discussed based on different point clouds in Sections 4.1 and 4.2. Finally, Section 5summarizes the main conclusions and gives an outlook for further development and possible use of the quality model. 2. State of the Art The surfaces of real objects are often represented as 3D point clouds after digitization. These 3D point clouds are an unsorted list of coordinates with additional (spectral) information. This representation is particularly well suited for measuring systems that use high-frequency scanning of object surfaces. Very efficient storage of single points or point groups (lines or arrays) is thus possible. This has caused the point cloud to become a quasi-standard for 3D object representations. The point cloud represents very efficiently, accurately and with a high resolution the geometry of scenes and objects. Unfortunately, with point clouds, the separation of individual objects is not possible right away. Thus, it is a necessary next processing step to derive information or models from point clouds. XLIII Remote Sens. 2022,14, 446 3 of 41 Current research on the separation of point clouds is mainly applied to autonomous operating systems, building modeling and computer vision (CV) tasks. Autonomous operating systems include autonomously driving cars, where information for obstacle avoidance, route planning and sign recognition has to be generated from the 3D point clouds [ 18 , 19 ]. CV and building modeling aim to enrich the point cloud with semantic information. The enriched point clouds are the basis for decision making and the creation of semantic models. If the point clouds represent complex scenes in which individual objects appear several times, then instancing is often the goal. Applications include the modeling of digital twins or the creation of city models, as well as the direct creation of simple building models based on point clouds and prior knowledge [20–22]. Different types of acquisition systems, segmentation tools and semantic point cloud datasets are available, forming the basis for the development of automatic point cloud separation methods. The application of these sets the quality of a semantic point cloud. A large amount of semantic training and benchmark point clouds are available. 2.1. Classification, Object Detection and Segmentation The definition of classification, object detection and segmentation is not clear in the literature, and these terms vary by research and application field. Different terms are used for the same separation task, or the meaning of the terms may be ambiguous. Some reviews [ 23 , 24 ] distinguish between classification, object detection and segmentation. Other researchers [ 25 ] use segmentation as an all-encompassing term for various categorization methods. To avoid misunderstandings, classification, object detection as well as semantic and instance segmentation are briefly defined below for this work. Classification: Classification is the assignment of a class feature (label) to one object. This can be a single point, a point cloud, a segment of a point cloud or another geometry type. Usually, semantic labels or IDs are assigned. The classification in the following is understood as the assignment of one semantic label to one point cloud segment. Object detection: In object detection, specific objects are defined based on geometric or spectral features in the point clouds. The individual object and not the entire point cloud is of interest, so that large parts of the point cloud are not evaluated in detail. Several objects in a point cloud can be detected and a unique identifier is obtained. Object detection is often used in conjunction with tracking objects in applications with multiple sub-point clouds. The objects are usually roughly described in terms of geometric size, position and orientation using bounding boxes. In other cases, it is not the objects as a whole that are of interest, but only certain surfaces or shapes [ 26 ]. These are searched for in the point clouds (shape detection). Semantic segmentation: The semantic segmentation has the goal of extending the features of the points by semantic labels. Semantic labels are semantic classes that usually describe real-world objects. The difference for the classification is that the segments are formed in this process step and a label is set for all points of the segment. A semantic segment can consist of several geometrically independent segments. For example, a point can belong to the class table; complementarily, it can belong to the subclass table leg. Moreover, the results of the classification of each point can form a new segment. Instance segmentation: An instance segment describes the geometric shape of one object. Instances in a point cloud can be distinguished by a unique identifier. An instance is usually enriched with semantic information. Points of the same semantic segment describe different objects. For example, if two tables are in one point cloud, then both carry the same semantic label. In order to distinguish the tables, instances must be created. Each table is an instance, which usually consists of a geometrically connected point cloud segment. The creation of a digital twin goes beyond this idea. For modeling a digital twin, new parametrized objects have to be formed that describe the point cloud content by generalizations such as a simple geometry. XLIV Remote Sens. 2022,14, 446 4 of 41 2.2. Captured and Synthetic Point Clouds Almost any semantic 3D point cloud is derived from a synthetic surface model or is captured by contactless sensors. An overview of the methods is given in Figure 2. Synthetic 3D point clouds are mostly generated from large collections of online model databases, such as [ 27 ]. These point clouds are generated efficiently by transforming a surface model into a regular or random point cloud. These points lie on the surface of the previous model or have synthetic noise added. Synthetic 3D point clouds usually represent only a single object or a small group of objects. Usually, they are used for algorithm development or prototype testing [28,29]. Figure 2. Capturing systems and basic data for the creation of 3D point clouds. Any acquisition technique for capturing reality has a certain resolution, precision and correctness, which can be found in the resulting point cloud. These point cloud characteristics often depend on the surface of the object, the acquisition distance, the environmental conditions and the measurement sensors. Optical sensors are the most widely used method for mapping reality. Optical sensors use light of different spectral bands to create a 3D point cloud of real environments with photogrammetric methods, as described in [ 30 ]. In particular, depth imaging cameras and LIDAR systems have been widely used in the last decade to create point cloud datasets [ 23 – 25 ]. The reasons are user friendliness, mainly moderate acquisition and evaluation costs [ 31 , 32 ] and the efficient capturing of larger areas. In addition to optical sensors, radar is sometimes used to create point clouds [33,34]. Depth imaging cameras consist of one or more cameras for different spectral ranges and an active emitter. Different principles for determining the image depths are used. For example, the Matterport Pro 3D Camera and the Microsoft Kinect V1 use structured light (SL) [35] and the Microsoft Kinect V2 uses the time of flight (ToF) method [36]. With the SL cameras, a monochrome near-infrared (NIR) image is captured in addition to a true-color image (red, green, blue (RGB)). The scene to be captured is illuminated by a projector with a known NIR pattern. The pattern consists of various bright and dark dots that are distributed in a non-correlating manner. The projected pattern is distorted by the geometry of the object. The depth is determined in several steps and for each pixel. First, the horizontal displacement of the dot pattern is determined based on the object distance. Based on the distortion, the depth of the respective pixel is then calculated in the next step using the equation for stereo triangulation [ 37 ]. For this purpose, the distortion in the unit of pixels, the base length (distance projector–camera) and the focal length in pixels are used. For each pixel, the distortion is determined using a local, e.g., 9 × 9pixel area, which is compared with a set of reference images for different depths. The comparison is performed using cross-correlation. An interpolation is performed between the highest XLV Remote Sens. 2022,14, 446 5 of 41 correlation values to increase up to sub-pixel resolution [ 36 , 38 ]. For further information on the SL method using the Microsoft Kinect V1 as an example, the reader is referred to [39]. Investigations of the Microsoft Kinect V1 show the precision expressed by the standard deviation (SD) of 1mm at 800mm distance and of 11mm at a distance of 3000mm [ 32 ]. According to [ 31 ], the correctness (offset to the target geometry) is up to 40mm for a captured distance of 1600mm (within a typical working range of 400 to 4000mm). Effects such as flying pixels (erroneous point measurement in a gap), color-dependent accuracy changes and multipath overlaps at edges do not or only occur at a very low level [ 31 ]. Moreover, for the Matterport Pro 3D Camera, which was used for online available training datasets by [ 40 , 41 ], the correctness, precision and resolution have been investigated in different studies. Here, a distance-dependent correctness of up to 80 mm for the furthest capturing distance was also determined. After a scaling factor is eliminated, a precision of better than 10mm SD can be determined for the entire working range [ 35 ]. A LIDAR point cloud was used as a reference for the mentioned study. The resolution of the Matterport Pro 3D Camera is 5 (horizontal) and 10 (vertical) points per degree [42]. The ToF technique is based on measuring the travel time of a signal from an emitter to reflect at an object’s surface and back to a receiver [ 30 ]. Pulse modulation (PM) and continuous-wave (CW) amplitude modulation are the most common ToF methods. In most depth imaging cameras, such as the Microsoft Kinect V2, CW amplitude modulation is used. In CW amplitude modulation, the object to be captured is continuously illuminated with NIR light, whose amplitude changes periodically. Because the signal needs a certain time between sensor and object, a phase shift occurs between the transmitted and received signal. This phase shift is proportional to the signal propagation time. If this time is multiplied with the known speed of light, the double distance between object and sensor system can be determined. The phase difference is determined for several modulated frequencies by correlating the received signal with the emitted reference frequencies. As long as the maximum distance is smaller than 2 π of the frequency, a distance can be determined as unique [36]. The precision of the Microsoft Kinect V2, as with the Microsoft Kinect V1, depends on the acquisition distance and varies between 1 and 3mm SD for the typical working range of 800to3000mm [ 31 , 32 ]. Recent depth imaging cameras, such as the Microsoft Azure Kinect, have a precision of less than 1mm for the same working range (static recording). Ref. [ 31 ] observed a constant offset of -18 mm for the whole working range of the Microsoft Kinect V2. Systematic erroneous measurements, such as flying pixels, color-dependent accuracy changes of up to 4mm, multipath-effects at edges of up to 30mm and a high dependence of distance measurements on temperature changes, are the disadvantages of this measurement principle [ 31 , 36 , 43 ]. These effects can be considered or eliminated in a later semantic segmentation. LIDARsystems are used for static and kinematic recordings of scenes. LIDAR systems emit a laser beam, which is projected onto a rotating mirror. Through the rotation, the beam is shifted by a certain increment. For each increment, the vertical and horizontal directions as well as the distance to the surface are registered. Together with the intensity value, and eventually with further spectral values, the 3D point cloud is created. For the distance measurements, there is the phase difference (PD) method, which can be used to realize a higher measuring frequency, and the PM method, which is less object surface-sensitive [ 30 ]. PM LIDAR systems are preferred for kinematic scanning on mobile platforms. Kinematic laser scanning usually involves measuring individual profiles, which are assembled as an entire point cloud using navigation data or algorithms, as in [ 44 ]. Mobile LIDAR systems are mainly used for outdoor applications and on robots. Medium-range LIDAR systems such as VelodyneHDL-64E are often used for creating datasets in research projects with a precision of 20mm [ 45 ]. High-end mobile mapping systems (MMS), such as the Riegl VMY-1 [ 46 ], allow the surveying of large-scale areas with a point accuracy of 15mm at 50m distance and a precision of 10mm. MMS such as the Nav Vis M6 are used in many studies [47]. XLVI Remote Sens. 2022,14, 446 6 of 41 The current state of the technology for indoor surveys includes terrestrial LIDAR systems (TLS), such as the Leica RTC 360,Z+F-Imager 5016 or Faro Fokus X 3D 330. These systems predominantly use the PD method and are used for distances shorter than 100m. Laboratory and field investigations show that, with these measuring systems, 3D point clouds with precision of less than1mm and correctness of less than2mm in the near field of up to 20m can be reached [ 48 ]. However, these values refer to optimal study circumstances such as matt or homogeneous surfaces. In practice, it has been shown for all LIDAR systems that the accuracy of the point clouds varies and scanning artifacts occur. Typical scanning artifacts are comet tails, mixed pixels on edges and multi-path effects on highly reflective surfaces, as shown in Figure 1. Other influencing variables, such as the measurement object, the setup and the environment, as well as the condition of the measurement systems [ 49 ], must be taken into account for the determination of the quality of a captured point cloud [ 50 – 52 ]. The resolution, the approximated accuracy, the acquisition method and the working range are crucial parameters that must be known or estimated for the later semantic segmentation of a point cloud. 2.3. 3D Point Cloud Datasets In various reviews [ 23 – 25 ] and in web databases (e.g., https://paperswithcode.com/ datasets accessed on 30 November 2021 and https://www.semanticscholar.org/ on 30 November 2021) on point cloud datasets and methods for point cloud processing, an overview of more than 100 publicly available point cloud datasets is given. These contributions summarize information on application areas, applied sensors, environmental circumstances or file formats. The main goal of these publications is to provide benchmarks for arithmetic evaluations. A semantic segmentation is not available for all existing datasets. A selection of semantic 3D point clouds is examined in more detail. The focus will be on the initial human segmentation and its evaluation. Not all datasets could be documented in the same level of detail. The datasets in Table 1were derived from synthetic surface models. All show one object of one known class. In some datasets, the object models are subdivided so that they can be used for semantic and instance segmentation. Since the point clouds are derived from synthetic models, the geometry can be considered free of scanning artifacts. However, errors can still occur during annotation and alignment. Table 1. Synthetic datasets with year of publication, data source, separation method (classification (Cls), semantic segmentation (SSeg) and instance segmentation (ISeg)), number of models, number of classes and environment. Dataset Year Data Source Separation Method No. of Models No. of Classes Environment ShapeNet [27] 2015 Trimble 3D Wareh., Yobi3D Cls, SSeg >220, 000 3135 In-/Outdoor ModelNet [53] 2015 Trimble 3D Wareh., Yobi3D Cls, ISeg 151,128 660 In-/Outdoor Shape2Motion [26] 2019 ShapeNet, Trimble 3D Wareh. ISeg Cls, SSeg 2440 45 In-/Outdoor An evaluation metric for classifications is introduced by the ShapeNet dataset, which describes how accurate or unique a classification is. Human annotators classify a semantic model until the classification accuracy varies by less than 2% [ 27 ]. The ModelNet dataset consists of 3D CAD models taken from web databases. The annotation is performed using Amerzone Mechanical Turk (AMT). The annotators classify different models using a web-based tool. For this, a model and a label are proposed. The annotators improve the correctness of a label for a displayed model by yes-or-no questions. An evaluation is conducted by the dataset designers for the ten most popular categories [ 53 ]. In the Shape2Motion dataset, a semantic segmentation of movable parts, such as wheels or car XLVII Remote Sens. 2022,14, 446 7 of 41 doors, and their properties is performed. An evaluation of the classification is carried out by simulating the motion directly after the segmentation and classification [26]. Complex point cloud simulation tools, such as the HELIOS++ [ 54 ] or Gazebo together with the Robotics Operation System [ 55 ], have reached a high level of development. These tools can be used to create point clouds from surface and CAD models that contain the characteristics of specific sensors and system configurations. Indoor datasets are commonly captured with depth imaging cameras. Some of the most popular datasets are summarized in Table 2. For a large number of datasets, depth imaging cameras are used in combination with an initial measurement unit (IMU). Together with the poses from the IMU and the images, a Simultaneous Localization and Mapping (SLAM) procedure is used to compute a multi-dimensional representation of the captured scene. The semantic annotation occurs either in images, videos, meshes or in 3D point clouds. Table 2. Indoor datasets recorded by depth cameras with year of publication, sensor, sensor method, separation method (classification (Cls), object detection (ObjD) and semantic segmentation (SSeg)), surface area and number of classes. Dataset Year Sensor Sensor Method Separation Method Surface Area Points No. of Classes SceneNN [56] 2016 Kinect v2 ToF Cls, SSeg 7078 m² 1,450,748 19 S3DIS [40] 2016 Matterport SL Cls, SSeg 6020 m² 12 ScanNet [57] 2017 Occipial (iPad) SL ObjD, SSeg 78,595 m² 17 Matterport3D [41] 2017 Matterport SL Cls, SSeg 219,399 m² 40 ScanObjectNN [58] 2019 SceneNN, ScanNet ToF, SL Cls, SSeg 2.971.648 15 The Stanford Large-Scale 3D Indoor Spaces (S3DIS) dataset is semantically segmented as a 3D point cloud using the software Cloud Compare (CC) [ 59 ]. For the SceneNN,ScanNet and Matterport3D datasets, a mesh is the segmentation base. All annotations are performed with custom tools. The SceneNN dataset is first automatically segmented coarsely and then finely. The graph-based segmentation algorithm of [ 60 ] is adapted and the segmentation is afterwards improved by the operator by separating, merging and re-forming the segments. The semantic annotation is performed by users attaching labels to the segments [ 56 , 61 ]. The semantic segmentation of the ScanNet dataset is performed by automatic pre-segmentation and a subsequent fine segmentation with classification using tools on AMT. In addition to semantic segmentation with meshes, CAD models are fitted into a mesh and are available as a different data format [ 57 ]. The Matterport3D dataset is semantically segmented in two stages and verified by ten experts. In the first stage, floor plans are derived using planes projected onto the mesh. In the second stage, the meshes of individual rooms resp. regions are segmented according to classes and instances using ScanNet’s tool [ 41 ]. For the ScanObjectNN dataset, the SceneNN and ScanNet meshes are the basis. A selection from this dataset is used and improved. Segments are rebuilt and categories are harmonized. A 3D point cloud with 1024 points is calculated out of each mesh. The verification of depth image datasets is mainly performed by experts or the authors [ 41 , 58 ]. Alternatively, the same dataset is semantically segmented by different people to identify error annotations [56]. No information is available about the validation of the S3DIS dataset [40]. A selection of recent semantic 3D point clouds generated with LIDAR systems is summarized in Table 3. These datasets will be used later in the quality model. Most 3D point clouds from LIDAR systems are for outdoor scenes and are captured with multi-sensor systems (MSS). With MSS, the capturing of larger areas is more efficient than with TLS. The geometric accuracy of a few centimeters, which is necessary for the majority of applications in geodesy and civil engineering, is maintained. In addition to the XLVIII Remote Sens. 2022,14, 446 8 of 41 LIDAR measurements, many MSS capture RGB images from the scanned scene to colorize the point cloud. Furthermore, these images can be used for semantic segmentation. The GT semantic segmentation of the datasets Paris-Lille 3D,Semantic3D,MLS1 TUM City Campus (MSL1 TUM CC), Toronto3D and Complex Scene Point Cloud (CSPC) is conducted completely or in parts with CC. For these datasets, the 3D point cloud format is the basis for data processing. This is also the case for the SemanticKITTI dataset, which is semantically segmented using a custom offline tool [ 13 ]. The Building Indoor Point Cloud (BIPC) dataset uses the LabelMe tool [ 62 ] for the segmentation and classification of 2D images. The 2D semantic segments are projected into 3D space after annotation. Any incorrect annotations in the point cloud are corrected using another 3D tool [ 63 ]. Another method to semantically segment 3D point clouds is to fit geometries, such as planes or boxes, into the point cloud. This is applied to parts of the dataset Semantic3D [ 15 ]. All points within a certain distance from the geometry are selected. The resulting segment is assigned to a class. Table 3. LIDAR-recorded datasets with year of publication, sensor, sensor method, separation method (semantic segmentation (SSeg) and instance segmentation (ISeg)), number of points, number of classes and environment. Dataset Year Sensor Sensor Method Separation Method No. of Points No. of Classes Environment Paris Lille 3D [64] 2018 Velod. HDL-32E MMS car SSeg 1431 M 50 Outdoor Semantic3D [15] 2017 Unknown TLS TLS SSeg 4 B 8 Outdoor SemanticKITTI [13] 2019 Velod. HDL-64E MMS car SSeg 4.5 B 28 Outdoor MSL1 TUM CC [14] 2020 Velod. HDL-64E MMS car SSeg, ISeg 1.7M 8 Outdoor Toronto3D [65] 2020 Teled. Opt. Mev. MMS car SSeg 78.3 M 8 Outdoor CSPC-Dataset [66] 2020 Velod. VLP-16 MMS backp. SSeg 68.3 M 6 Outdoor BIPC-Dataset [63] 2021 Velod. VLP-16 MMS backp. SSeg - 30 Indoor Closely related to the semantic segmentation is its evaluation. The Semantic3D dataset is evaluated by class comparisons in the overlapping areas of the neighboring point clouds. For this purpose, all points in the neighborhood of an adjacent point cloud are selected from a given point with a search radius of 50mm. The classes of the selected points are compared with the class of the initial point [ 15 ]. The SemanticKITTI and the CSPC datasets are evaluated and improved by experts in a second processing step [ 13 , 66 ]. For the BIPC dataset, the segments created in 2D are evaluated on the 3D point cloud [ 63 ]. Statistical evaluation of semantic accuracy for all datasets is not documented. No information on the verification of semantic segmentation is available for the Paris-Lille 3D,MLS1 TUM CC and Toronto3D datasets. Based on the datasets from the last six years, it can be concluded that more and more LIDAR systems are being used. Mainly LIDAR datasets of outdoor areas are created, because of the larger range and the higher resolution of these systems. For indoors, depth imaging cameras are still commonly used. Since many of these data come from the CV domain, surface models or voxels are additional output formats, along with point clouds and images. It can be seen that the datasets are not necessarily larger in terms of classes and points, but the annotation is more specialized and improved compared to early datasets. Earlier datasets are evaluated with new tools and optimized for specific tasks. The manual annotation can be still identified as a bottleneck. 2.4. Point Cloud Annotation Tools Many annotation services and tools are used for autonomous driving or driver assistance. For this application, a few outdoor classes need to be (roughly) annotated. An overview and comparison of 33 annotation tools for this area of application is presented in [ 67 ]. These annotation tools mainly use simple geometries, such as bounding boxes, XLIX Remote Sens. 2022,14, 446 15 of 41 in the required quantity. It is the basis of all further characteristics and must be fulfilled in order to carry out a semantic segmentation and its evaluation. The process reliability describes how the process was carried out. This characteristic can be determined by performing the semantic segmentation several times. After each segmentation, accuracy parameters are determined, which can be used as terminating criteria, as in [ 27 ]. Other variants require that a certain number of iterations must be fulfilled in order to determine a quality parameter. This variant is preferred for the description of the process quality, since it maps the variance of the metric and parameter values. Using this, the achievable performance can be determined by a process setting. Due to the complexity of these tasks, the average repetition factor to determine this parameter is usually very small, as shown in [13] factor 2, in [57] factor 2.3 and in [68] factor 4. Completeness gives the degree to which the necessary information, determinations and execution of the work steps for the classification are present. Consistency is the degree to which the measured values match the data model. Here, it is necessary to check whether the point cloud features are present and whether they take the corresponding range of values. For the semantic segmentation, correctness , precision , and semantic accuracy are different characteristics that have different underlying causes and different effects on the usability of the point cloud. Moreover, these characteristics are often defined and summarized in different ways. For example, if only the performance of a semantic segmentation method is to be considered, correctness and precision are often combined with accuracy. The accuracy is commonly given when ML and AI algorithms are used. In many semantic segmentation applications, this defined characteristic is expressed by the parameter IoU, also known as the Jaccard index, or the F1 score, also known as Dice’s index. These parameters are the weighted averages of both characteristics. It is advantageous to apply one combined characteristic of accuracy and its meaningful parameter, e.g., IoU, for better comparison. Other applications use more than one parameter to describe the different perspectives of accuracy for a more distinguished and cause-oriented view. For the analysis of the semantic segmentation process, two types of errors are possible: a point is erroneously assigned to a class to which it does not belong or a true point of this class is not recognized as member of it. These two errors are known as firstand second-type errors from statistical tests [ 87 ]. The segmentation precision can be considered an error of the first type. This error specifies how well an annotator or an algorithm can distinguish classes—for example, how accurately class boundaries can be drawn. The segmentation correctness can also be considered a second-type error. This error describes how well a class can be recognized, e.g., how unique the point features are. Thus, the best features are used to obtain a class of homogenous points. This type of error can be of importance depending on the analysis in question. For example, it may be less critical if not all points of a large class (such as floor) are detected during semantic segmentation, as long as these points are not classified or assigned to a class (e.g., scanning artifacts) that is not further used. More problematic are additional points (e.g., from scanning artifacts) that are assigned to the class floor, because the point cloud represents incorrect semantics. Thus far, correctness and precision based on the number of points describe the quality of a semantic point cloud. However, these characteristics do not give any information about the geometry of the semantic classes and its geometric size changes due to errors. In order to be able to evaluate the geometric aspect as well, the characteristics of correctness and precision have to be extended. Geometric correctness can be determined if a (dense) reference point cloud or surface model is available. The correctness can be determined for each individual point. This information can no longer be evaluated for several hundred thousand points. The correctness of a point cloud can be determined by the mean, average deviation or standard deviation of all points in a segment. In general, correctness is the degree to which the abstract model matches the achieved semantic segmentation result. This can be divided into user-dependent and software-dependent correctness. The user-dependent correctness is based on the understanding of the CD and the usage of the LVI Remote Sens. 2022,14, 446 16 of 41 software by the user. The software-dependent correctness refers to errors in the software (e.g., incorrect parameters or programming). However, a separation is only possible if the semantic segmentation is carried out several times under controllable conditions. The quality characteristic of precision is described by the parameters for the semantic and geometric precision. The term "precision" should be defined clearly, because there are different definitions in use. In the geodetic context, precision is often understood as repeatability [ 87 ]. The deviation of the results of an experiment to its mean value after n repetitions is determined. For the semantic segmentation process, this definition would lead to the determination of how much the individual segmentations deviate from each other. This shall not be the main subject of the investigation, since a deviation to the mean of several segmentations usually has no relevance for a practical application. Nevertheless, it makes sense to repeat a segmentation and to calculate a joint point cloud from these repetitions in order to increase the reliability, as mentioned above. Usually, deviation from a reference point cloud is required. This can be expressed by the ratio of true points to all points assigned to a class [ 88 ]. This term describes how much of the segmentation is “correct” and is commonly used in ML. Mostly, the inverse proportion is of major importance for the development of an application, because this describes what does not work yet [ 89 ]. This proportion is then the subject of analysis. In addition to the use of the number of points, it is advantageous for 3D models and point clouds to also use the areal ratios as well as geometric parameters. The semantic accuracy describes how well the semantic label fits to a semantic point cloud segment. The difficulty is in defining what is semantically correct, which attributes are described and which depth of description and distinction must be applied. For the definition of what is semantically correct, no universally valid definition can be found. An attempt to standardize this problem was discussed in Section 3.2. For the type of attribute description, the IFC standard [ 80 ] can be used. This is designed for the development and not for the documentation. This can be explained using the example with the tables. The table itself forms a semantic class. These classes can be differentiated during the next stage into a frame and table top. As far as we know, there is no standardized scheme for this definition, so that an individual CD as shown in Appendix Amust be developed and applied. 3.4.2. Quality Parameters The seven quality characteristics used for semantic segmentation can be described by quality parameters. These parameters describe the property that an object has for a certain characteristics. For instance, these parameters are the presence of a certain data format as a qualitative parameter or the number of points (NoP) as a quantitative parameter. This will be demonstrated in an example in Section 4.1. The evaluation of point clouds by the quality model will be covered in Section 4.2. For the evaluation, the quality parameters must be determined and threshold values must be set. Furthermore, the parameters for the semantic segmentation can be distinguished into parameters with object relation (O), concerning the point cloud, and process relation (P), such as the time required for an action or the use of a certain CD. All parameters for the semantic segmentation task are briefly explained and shown in Tables 5–11. The parameters are numbered in the text and refer to the corresponding table entry with P#.# for a clear understanding. Quality parameters for characteristic availability describe which information must be available about the process and the point cloud for a description and an evaluation (Table 5). These parameters are the abstract model expressed by the CD (P1.1), the size of the point cloud expressed by the NoP (P1.2) and the area size (P1.3), as well as the object features (e.g., x-, y-, z-coordinates) before (P1.4) and after (P1.5) the semantic segmentation. Furthermore, the file format output (P1.6) and use restrictions (P1.7) must be investigated. The use restrictions refer to the question of whether a dataset can be used for an application or processing step. Further restrictions are that certain datasets may not be used for training. LVII Remote Sens. 2022,14, 446 17 of 41 The parameter P1.7 ensures an objective evaluation of the datasets. Thus, it is considered that any dataset has a certain bias, which is learned by ML algorithms [86]. Table 5. Parameters for availability. P. No. Parameter Name Unit Range P/O P1 Availability P1.1 CD exists yes/no P P1.2 Number of points >0 O P1.3 Area size m2>0 O P1.4 Object charac. in yes/no O P1.5 Object charac. out. yes/no O P1.6 File format out e.g., pts O P1.7 Use restriction yes/no O Table 6. Parameters for process reliability. P. No. Parameter Name Unit Range P/O P2 Reliability of Process P2.1 Number of segmentations >1 P P2.2 Average time required % 0–100 P The parameters number of segmentations (NoS) (P2.1) and average time required (ATR) (P2.2) describe the reliability of the process (Table 6). If a point cloud is independently semantically segmented more than once, the reliability can be measured. The more frequently a process is carried out, the more reliable are the correctness and accuracy. This is the theoretical assumption. The parameter NoS describes how often a segmentation was performed with a certain method. It is the basis for the calculation of other parameters and can also be used as a quality measure. The ATR can be used to compare different semantic segmentation methods. The ATR is calculated for each method. The average time of all annotators with any method is of interest. The maximum segmentation time of all methods is the value ∆tmax . The ATR is calculated from Equation (1), where i stands for the respective segmentation. ∆ti is therefore the time needed for the segmentation i . Moreover, the user-dependent segmentation time can be analyzed if all segmentations performed with a certain tool are compared. The parameter ATR describes the process and allows the planning of the working time. ATR =∑imax i=1k∆ti∗100 ∆tmax k i(1) Table 7. Parameters for completeness. P. No. Parameter Name Unit Range P/O P3 Completeness P3.1 Semantic segmentation rate % 0–100 O P3.2 Number of classes >0 O The completeness of a semantically segmented point cloud (Table 7) is described by the semantic segmentation rate (SSR) (P3.1) and number of classes (NoC) (P3.2). The parameter SSR describes how many points have been assigned to any class. The SSR is the quotient of the number of classified points (Pcls) and all points (Pall) (Equation(2)). SSR =Pcls Pall ∗100 (2) A point cloud that is only segmented in parts often occurs in the application phase. The semantically segmented parts of the point cloud are used for training or for the evaluation LVIII Remote Sens. 2022,14, 446 18 of 41 of an algorithm. The rest of the data are then semantically segmented using the automatic method. The parameter NoC describes how many classes are available for a certain dataset. Table 8. Parameters for consistency. P. No. Parameter Name Unit Range P/O P4 Consistency P4.1 Geometric Consistency (GC) of x, y, z m ≥0 O P4.2 Spectral Consistency of RGB (SCRGB) 0–255 O P4.3 Spectral Consistency of I (SCI) 0–255 O P4.4 Class equality 0–1 O The consistency of the data (Table 8) is determined by the units and the scaling ranges of the object features (P4.1 to P4.3). Each object parameter directly relates to a quality parameter. The determination can be achieved automatically or taken from the data (e.g., using a text editor). Furthermore, the consistency is described by the measure of the class equality (CE) (P4.4). This is calculated from the target value of a balanced class distribution ( Ctarget ). All classes should be represented by the same amount of points, so that, later, an ML procedure has optimal learning conditions. However, this requirement is never given with real datasets, because classes such as walls and floors are overrepresented by points. The proportion of points of a class in relation to the total NoP is expressed by a ratio in the value range 0–1. The actual distributions are then calculated ( Cact ). The differences between the target and actual values for each class are determined. The sum of the absolute differences divided by two is a measure of balance (Equation(3)), where 0 represents a balanced ratio and 1 an unbalanced ratio. CE =∑k i=1k(Ctarget −Cact)k 2(3) Table 9. Parameters for correctness. P. No. Parameter Unit Range P/O P5 Correctness P5.1 Recall of points class x % 0–100 O P5.2 Recall of area class x % 0–100 O The correctness (Table 9) of the semantic segmentation can be described by the parameter recall of points (RP) (P5.1). The PR is the rate between correctly assigned true positive (TP) points and the NoP in the abstract model for a certain class (TP and false negative (FN) points) (Figure 7). It is expressed by Equation (4). This parameter depends on the size differences of the class in the abstract model. If the classes differ greatly, as can be evaluated by the parameter CE, a comparison of different classes may lose significance. For a small set, even a few FN points can significantly lower the parameter. This problem is discussed in [ 89 ] and described by a new parameter for informativeness. For applications in the context of point clouds, this parameter is unsuitable due to the irregular distribution of the points. RP =TP TP +FN (4) RA =TParea TParea +FNarea (5) To avoid the point cloud density problem, the representation in the form of areas can be used. Here, the areas are calculated for the point cloud segments. Instead of the NoP, the TP area size can be inserted into Equation (4). The result is the recall of area (RA) in Equation (5). The correctness is now described by the area that is covered by TP points LIX Remote Sens. 2022,14, 446 19 of 41 divided by the area of all reference points of this class. As an intermediate step to calculate these parameters, the areas that are correctly and incorrectly assigned are calculated. In the case of incorrect assignments, the distinction between FN and FP areas is of interest. The parameter RA expresses the influence of FN surfaces. The influence of the false positive (FP) areas is described in the following, among others, by the precision of area (PA). The FN and FP points are visualized in Figure 8. This visualization allows an analysis of the semantic segmentation, e.g., the assignment of scanning artifacts to a class or the occurrence of classification gaps can be determined. Figure 7. Schematic representation of the confusion matrix for the floor class with entries for TP, FN, FP and true negative (TN) points. Table 10. Parameters for precision. P. No. Parameter Unit Range P/O P6 Precision P6.1 Precision class x % 0–100 O P6.2 Precision area class x % 0–100 O P6.3 MD of FP pts. class x mm ≥0 O P6.4 SD of FP pts. class x mm ≥0 O The precision is expressed by the precision of points (PP) (P6.1) and the PA (P6.2). The PP is the ratio of TP points of a class to all points assigned by the segmentation of this class (Equation (6)). The assigned points could also be expressed as the sum of the TP and the FN points (Figure 7). PP =TP TP +FP (6) PA =TParea TParea +FParea (7) The consideration of the characteristic precision based on areas that are spanned by the point cloud segments can be advantageous when using the point cloud as a model. For LX Remote Sens. 2022,14, 446 20 of 41 a geometric expression, Equation (7) can be used to determine PA. The visualization of the FP points is given in Figure 8, which is a good starting point for the analysis process. Figure 8. Segmented point cloud of the class table colored by TP, FP and FN points. Thegeometricpartoftheprecisioncanalsobedescribed bytheparametersmaximumdeviation (MD) ofFPpoints (P6.3) and SDofFPpoints (P6.4). The MDofFP and SDofFPpoints rely on the FP points of the semantic segmentation. They are the points that change the geometry of the semantic class, as shown in Figure 9. For this consideration, only classes with semantic objects are considered, since, normally, the goal of semantic segmentation is to extract objects and to remove scanning artifacts. The geometric deviation of the point cloud segment is of major importance for creating a model. If the point cloud is used to create a mesh, then the MD, which is the enlargement of the class segment, is decisive. This is expressed by the furthest FP point. For modeling on the basis of point clouds or the representation of the recorded objects by symbols, as is the case at the LoD 100 for a BIM application [90], the parameter SD of FP points is more meaningful. The semantic accuracy (Table 11) is described by parameters that can be expressed by yes-or-no questions. Documentation of the process and visual inspections can be used to determine the CD applied parameter (P7.1) and whether it is structured hierarchically (P7.2). The parameter CD applied can be answered with yes if the CD is used and at least one class is segmented. The parameter Hierarchical CD can be confirmed if the used CD has several levels (at least two) and so different semantic detailing levels are available. The query whose class was finally used is expressed by the parameter P7.3. If the class is present and semantically correct, the parameter is answered with yes. Table 11. Parameters for semantic accuracy. P No. Parameter Unit Range P/O P7 Semantic Accuracy P7.1 CD applied yes/no P P7.2 Hierarchical CD yes/no O P7.3 class x used yes/no O 3.4.3. Descriptive and Evaluative Function A quality model such as the one above can have two functions. One is descriptive and the other is evaluative, as described by ISO 9000 (2015) [73]. LXI Remote Sens. 2022,14, 446 21 of 41 For the descriptive use , the aim is to display and analyze how individual parameters (defined as significant by the model) vary when influences change. Different settings, tools or work processes for a semantic segmentation can be compared. Quality parameters are not transformed into another representation or range for this purpose. The main influencing characteristics for the development of a semantic segmentation process are considered and this is one main application of the quality model. More precisely, the influence of the initial (manual) segmentation of a point cloud is investigated. Thus, the model also provides the basis for describing an automatic (e.g., ML-based) semantic segmentation process, as considered in many works, such as [91–94]. Figure 9. Calculation of the SD of FP points σ on the example of a chair. Green TP points are within the object boundaries. The red FP points were added to the chair class but actually belong to another class. For the evaluative use , the suitability of a point cloud for an application should be assessed. It should be derived from the parameters whether a point cloud in combination with the segmentation method is suitable for a certain application or not. For this purpose, the calculated parameters of the quality model are crucial. An example application would be to use a semantic point cloud to determine the wall surface area, to calculate the renovation costs, based on the as-built wall surface area. For this task, correct semantic segmentation is crucial. The point cloud should be evaluated by applying a quality model in advance. The quality of the individual parameters must be defined by limit or target values. These values are derived from the application. The evaluation steps are defined according to the scheme shown in Figure 10. After the limit or target values have been defined, they are compared with the determined actual values. This adjustment can be represented in an automatic procedure by one Boolean value. In the simplest overall evaluation method, all parameters must be true for sufficient quality. The weighting of the parameters for special cases prevents excessively rigorous filtering. The central issue is the limit or target values, which are not always known and have to be estimated based on experience. Figure 10. Evaluation of the suitability of a point cloud with the quality model. LXII Remote Sens. 2022,14, 446 22 of 41 4. Applying the Quality Model The benefit of the quality model as a basis for describing and evaluating the properties of a semantic 3D point cloud will now be explained by some examples. The performance of the quality model is shown on the basis of two of our own indoor point clouds and other publicly available point cloud datasets. Our own point clouds are shown in Figure 11 and were semantically segmented independently, multiple times, using two different semantic segmentation tools. The quality of the point cloud and the semantic segmentation process are described by the quality parameters. The evaluation performance of the quality model is considered for our own and the publicly available datasets. The applications of interest are the analysis of: • Semantic point cloud as a model; • Semantic point cloud as a modeling basis; • Semantic point cloud as training data. Target values are defined in each case. The geometric, semantic and formal characteristics of the point cloud are processed and used for an application. However, these point cloud characteristics have a degree of uncertainty if the semantic point cloud was created by capturing a real object and performing a semantic segmentation. The possible errors and the quantitative uncertainty of the sensors are described in Section 2.2. It can be stated that the usually resulting effects of currently available and used sensors do not significantly affect the indoor modeling applications. Our own point clouds were recorded with the Z+F Imager 5016 using a resolution of 6mm at 10 m. The quality was set to high to reduce the noise while still having a moderate (in practice useful) recording time of 6min [ 95 ]. The geometric correctness of this point cloud on a flat surface can be estimated as 2mm to 3mm according to the investigation of [ 48 ], using the DVW-test-field-method according to [ 96 ]. This accuracy varies due to the different surface shapes and other object properties. In addition, scanning artifacts occur, as shown in Figure 1and described in Section 2.2. The focus is now on the semantic segmentation, where errors are caused by tool settings and the annotator. Figure 11. Points to be examined without semantic segmentation. Objects of the chair, table and floor classes, as well as scanning artifacts, are shown. The point clouds in Figure 11 are very challenging for semantic segmentation. A CD was developed and applied in order to investigate segmentation problems. This CD consists of five classes and is partly hierarchically structured. The classes of the first level are floor, furniture and scanning artifacts. In the second level, the furniture class LXIII Remote Sens. 2022,14, 446 23 of 41 is divided into table and chair. Tables and chairs are two object classes that are spatially and geometrically similar, which makes segmentation difficult. The points of these two classes also have similar spectral properties. Finally, the object surfaces are highly reflective and the geometric shape is susceptible to the occurrence of scanning artifacts. The floor class was integrated to simulate scenic segmentation with foreground and background objects. The separation of scanning artifacts is a complex task, even for humans, where subjective decisions must be made and learned. The test point cloud does not represent any real particular task, but is intended to demonstrate achievable performance on challenging cases. The point clouds show recordings of a laboratory (Lab) and a seminar room (Room), which were automatically segmented with the PCCT using the spectral parameters color and intensity. The point clouds were processed by up to nine different annotators. These are the test point clouds Lab RGB,Lab I as well as Room RGB and Room I. Furthermore, the point clouds Lab and Room were processed with Recap, in which the annotators determine the segments by themselves. These are the datasets Lab R and Room R. 4.1. Quality Model to Describe Semantic Point Clouds The description of a semantic point cloud and a segmentation process is always based on a selection of characteristics, with the goal of being able to answer a specific research or practical question. The research question for the following consideration is: What influence do the segmentation tool and different annotations have on the quality of the semantic segmented point cloud? The motivation for this question is to develop an efficient, effective and traceable segmentation process. Different experimental settings and development stages shall be described, so that their influences on the process can be analyzed. This should also result in more convenient point clouds for models and training data, as well as improved process and algorithm understanding. All characteristics of the model are described in detail in the following. 4.1.1. Reliability Characteristics The reliability of a point cloud can be described mainly by formal information or metadata, as listed in Table 12. The creation and use of a CD, which regulates which objects will be segmented and classified, is of primary importance. A comparison of semantic segmentation is only possible if the CD is kept constant. The parameter CD exists must be available to utilize all other semantic-based descriptions. The accuracy of the implementation of the CD is described by the parameters of semantic accuracy in Section 4.1.3. For the test point clouds, a CD exists, which describes the semantic classes of floor, furniture, chair and table, and scanning artifacts. The size of the point cloud is another formal parameter, which is described by the NoP and the surface area. The NoP that can be processed by segmentation tools varies widely. Sometimes, the point cloud is automatically reduced to a maximum NoP. This filtering changes the point cloud structure and, depending on the application, can result in unwanted effects, such as the loss of surface details. The Lab and Room point clouds consist of 2.7 and 14.5 million points. The surface area of the objects covered by points is 51 m 2 and 61 m 2 for the Lab and the Room point clouds, respectively. Based on these two parameters, an additional useful parameter, the average point cloud density, can be calculated. The average point cloud density can be used as the resolution of the point cloud. This varies with the distance to the recording device, and this shows that the parameters of the quality model are chosen to be fundamental, so that optional parameter extensions are possible. The segmentation tools require certain point cloud features to enable processing. Spectral features are often used to perform an automatic segmentation or to color the point for better visual differentiation. Most point clouds have geometric features (coordinates) and spectral features for color and intensity. In addition to these features, normals (N) are calculated to create perspective images or orient the single point within their neighborhood, as done with the PCCT. These features can enrich the point cloud after the semantic LXIV Remote Sens. 2022,14, 446 24 of 41 segmentation. The exported feature can change during the semantic segmentation. The point clouds in the example are only extended by the feature semantic class. This is expressed by exporting each class as a single pts file. Closely related to the feature parameters of the point cloud is the file format that is available for import and export for software. The pts format is supported by all tools being used. This file format corresponds to the data model of Section 3.3. The used data model states that all segments should be available as an individual file. If the data model requires that the semantics of the point cloud have to be included in one file, then a different export file format must be used. This file format must have one additional space for the semantic label. The point clouds Lab and Room are currently not licensed and are only used internally, so there is no restriction on usage (P1.7). This means that the use of the datasets cannot be traced. Table 12. Calculated and determined values for the quality parameters of availability and reliability of process. Object parameters with * are calculated in the segmentation software. P. No. Parameter Name Lab RGB Lab I Lab R Room RGB Room I Room R P1 Availability P1.1 CD exists yes yes yes yes yes yes P1.2 NoP 2,790,352 points 14,526,242 points P1.3 Area size 51m261 m2 P1.4 Object char. in. x, y, z, I, R, G, B, xN*, yN*, zN* P1.5 Object char. out. x, y, z, I, R, G, B, Class P1.6 File format out. pts/csv pts/csv pts pts/csv pts /csv pts P1.7 Use restriction no no no no no no P2 Reliability of Process P2.1 NoS 7 7 9 8 8 8 P2.2 ATR 13% 13% 55% 38% 45% 49% In addition to point cloud metadata, metadata about the process are also represented by the process reliability, as shown in Table 12. Reliability can be determined if a process is performed independently multiple times. It can be determined by observing which parameters change systematically and which are random. According to the research question, two influences should be analyzed. On the one hand, the influence of different users is considered, and on the other hand, that of different tools is assessed. The repeat accuracy of different users is investigated in Section 4.1.4. At this point, the focus is on the two different tools. For a statistical consideration, the number of seven to nine annotations per tool is too small. However, a qualitative or comparative description of the influences of the tools in the form of a tendency is possible despite the small number of samples. For this purpose, the following values are not based on the annotations of individual annotators, but on a joint point cloud with all annotations. For the determination of the parameters of the datasets Lab RGB and Lab I, seven different annotations were performed; for the Lab R dataset, nine annotations were performed, and for the datasets Room RGB,Room I and Room R, eight annotations were performed. The ATR is calculated based on the longest time for semantic segmentation for each point cloud. The maximum time is 120 min for the point cloud Lab and 194 min for the point cloud Room. For both point clouds, the semantic segmentation with Recap takes the longest. The ATR values in Table 12 show that the PCCT provides an average of only 13% of the maximum time for small point clouds such as Lab. With Recap, the ATR is 55% for the Lab point cloud. For the larger dataset, it can be seen that the PCCT can be used to work faster on average, but the differences in time decrease with increasing point cloud size. The parameters of availability and process reliability are the basis on which to describe further parameters that have a more practical meaning for the investigated question. Thus far, it is described how a process can be carried out with the selected data and resources, how reliable this process and the other quality parameters are, as well as how efficient the tools and its usage are in comparison to others. LXV