Semi-automated extraction of information from large-scale historical maps

Semi-automated extraction of information from large-scale historical maps Dissertation submitted in partial fulfillment of the requirements for the degree of Doktor-Ingenieur (Dr.-Ing.) at the HafenCity University Hamburg in the field of Geodesy and Geoinformatics by Inga Schlegel Supervisors: Prof. Dr.-Ing. habil. Jochen Schiewe Prof. Dr. Lorenz Hurni Hamburg, March 2023 License Notice This is an open access publication distributed under the terms and conditions of the Creative Commons Attribution (CC BY 4.0) license (https://creativecommons.org/licenses/by/4.0). “We cannot predict the joys and sorrows that await us on this journey through life. Courage is our only map.” Tess Thompson To my mom. V Acknowledgements At this point I would like to thank these people without whom this journey would have been much more difficult. First and foremost, my deepest appreciation goes to my doctoral advisor Jochen Schiewe for his unwavering support throughout the constitution of this thesis. Thank you for always having an open ear, for generously sharing your wisdom and knowledge and being sensible and patient. Your motivation and confidence always made me leave our fruitful discussions with a pleasant feeling. Many thanks to Lorenz Hurni for reviewing this thesis as second supervisor. Your consultation offered me a better understanding of the overall picture. I am fortunate to have been part of the g2lab working group. Numerous profound and less profound conversations with Martin Knura, Güren Dinga, and Patrick Postert made my working environment more than pleasant. I really enjoyed our regular espresso chats. I am very grateful that I was able to share my office with Martin who was constantly willing to help me out with major and minor research problems and kindly considered my lack of interest in soccer topics. You all were a great company! Without the unconditional devotion of my proofreaders, this accomplishment would not have been possible. Therefore, I am deeply indebted to Annette Scheider and Maike Schotten. Annette for even sacrificing her well-deserved vacation and Maike for “being proud” and such a good and honest friend. I also had great pleasure of working with my past and present colleagues Hannes Kröger, Bonan Wei, Laura Jensen, Simon Deggim, Eike Barnefske, and Anni Wisbar. You all helped me in your own ways to keep motivated for finishing this thesis. Thank you for your professional, personal, and moral support and for refreshing conversations that went beyond the scientific scope. Lukas, I would not have been able to finish this thesis without your dedicated support and patience. Thank you for always encouraging me and having the time for proof-reading my works over and over. I was continually amazed about the great effect of your semi-distant view of the matter. I thank you with all my heart. I am also grateful to my dear friends Svenja, Anja, Nona, and Maria for sharing my excitement and cheering me particularly in the final stage of getting it done. Finally, I would like to thank my parents and my brother for their unceasing encouragement, silent contributions, and for showing me what is truly important in life. My mom, in particular, gave me the necessary strength even in difficult times and brought me back down to earth time and again. VII Abstract Historical maps are important relics to reconstruct our past. New insights and information can be unveiled and make long-term morphological developments of different spatial environments understandable. As part of the investigation of urban areas, dynamics of settlements such as transformations of built-up areas or changes in road networks are of particular interest. However, detailed geographic information concerning urban history is way more accessible from large-scale historical maps than from other sources. Due to the great number and visual variety of available historical maps and the lack of appropriate tools, researchers still often revert to laborious manual means in the analysis and comparison of these. This thesis provides a comprehensive solution to semi-automatically unlock and retrieve geometrical as well as textual content from large-scale historical maps. Thus, the spatiotemporal exploration of a city’s individual buildings, roads, or water areas can be considerably improved. Several shortcomings in this research field are addressed in this thesis. It is the first study to present a holistic concept for semi-automated extraction of geometric and semantic content from large-scale historical maps. Needs of users of historical maps are identified and evaluated in terms of a conducted user survey. The developed and demonstrated workflow is able to extract shapes of discrete map objects representing real-world equivalents as well as their labels. In addition, this thesis considers further processing of the extracted information: To be usable in geographic information systems, map objects are vectorized and labels are provided in the form of text strings. Spatial referencing creates the foundation to manage and store deduced data in databases and to assign additional knowledge. Therefore, an improved starting point for the comparison of historical maps with other geodata is provided. The developed workflow is applicable to comparable, typically monochrome, large-scale historical maps of similar complexity to the sample used for this thesis. The central question this research pursues is how the extraction of information from large-scale historical maps can be facilitated to render them searchable, analyzable, and comparable with other maps. It is shown how objects and labels from simple scans of historical maps can be transferred into machine-readable data. With the help of object-based approaches, single map objects can be identified and differentiated based on the model of human perception, i.e., by means of various visual variables such as color, texture, and shape. Available tools for detecting and recognizing labels are used and amended with additional enhancements identified and developed for this thesis. Finally, further methodologies, e.g., from image processing, help to develop a novel and comprehensive approach for the extraction of information from large-scale historical maps. The involved processes benefit from each other and reduce human interaction and subjectivity, time, and labor to a necessary minimum. As maps were and are still made to be viewed and interpreted by humans, automated methods taking into consideration principles of human perception generally achieve optimum results. Providing editable vector data of historical maps considerably contributes to their processability, analyzability, and comparability and thereby facilitates the daily work of historians, librarians, or urban researchers. An additional allocation of related semantic information allows users to search for keywords, juxtapose e.g., names of streets or measures of buildings, or simply analyze their persistence over time. In conclusion, this thesis demonstrates the efficiency of comprehensive workflows for semi-automated information extraction from large-scale historical maps. It contributes to an improved transmission and perception of geographic information. By facilitating the comparison of urban geospatial data representing different times, spatiotemporal changes and developments in human history become more clearly recognizable. IX Zusammenfassung Historische Karten sind wichtige Zeugnisse zur Rekonstruktion unserer Vergangenheit. Neue Erkenntnisse und Informationen sowie langfristige morphologische Entwicklungen verschiedener Teilräume können sichtbar und nachvollziehbar gemacht werden. Im Rahmen der Erforschung urbaner Strukturen sind Siedlungsdynamiken wie Veränderungen von bebauten Gebieten oder von Straßennetzwerken von besonderem Interesse. Mittels großmaßstäbiger historischer Karten sind detaillierte geographische Informationen zur Geschichte einer Stadt oft greifbarer als aus anderen Quellen. Aufgrund der großen Anzahl und visuellen Vielfalt historischer Karten sowie fehlender Tools greifen Forschende bei der Analyse und dem Vergleich dieser Karten noch immer auf mühsame manuelle Verfahren zurück. Diese Dissertation bietet einen umfassenden Lösungsansatz für die halbautomatisierte Extrahierung von geometrischen und semantischen Inhalten aus großmaßstäbigen historischen Karten. So wird die raumzeitliche Untersuchung einzelner Gebäude, Straßenzüge oder Wasserflächen einer Stadt erheblich verbessert. Diese Arbeit befasst sich mit verschiedenen Defiziten innerhalb dieses Forschungsbereichs. Erstmalig wird ein holistisches Konzept für solch eine halbautomatisierte Extrahierung vorgestellt. Anhand einer Nutzerstudie werden Anforderungen an historische Karten ermittelt und evaluiert. Der demonstrierte Workflow ist in der Lage, diskrete Kartenobjekte, die reale Pendants darstellen, sowie deren Beschriftungen zu extrahieren. Darüber hinaus wird in dieser Arbeit die Weiterverarbeitung der extrahierten Informationen betrachtet: Kartenobjekte werden vektorisiert und Labels in Form von Textstrings bereitgestellt, um sie in Geographischen Informationssystemen nutzbar zu machen. Eine räumliche Referenzierung bietet eine Grundlage, um abgeleitete Daten in Datenbanken zu speichern und zu verwalten und um zusätzliche Informationen zuzuweisen. Damit wird eine verbesserte Ausgangslage für den Vergleich von historischen Karten mit anderen Geodaten geschaffen. Der entwickelte Workflow ist auf vergleichbare, in der Regel monochrome, großmaßstäbige historische Karten von ähnlicher Komplexität anwendbar. Wie die Informationsextraktion aus großmaßstäbigen historischen Karten erleichtert werden kann, um diese durchsuchbar, analysierbar und mit anderen Karten vergleichbar zu machen, ist zentrale Frage dieser Arbeit. Es wird aufgezeigt, wie Objekte und Labels aus einfachen Scans historischer Karten maschinenlesbar gemacht werden können. Mithilfe objektbasierter Ansätze können einzelne Kartenobjekte anhand verschiedener visueller Variablen wie Farbe, Textur und Form identifiziert und differenziert werden. Etablierte Prozesse zur Erkennung von Labels werden angewandt und weiter verbessert. Der neuartige und umfassende Ansatz für die Informationsextraktion aus großmaßstäbigen historischen Karten wird durch zusätzliche Methoden, beispielsweise aus der Bildverarbeitung, ergänzt. Die implementierten Prozesse begünstigen einander und reduzieren die menschliche Interaktion und Subjektivität, Zeit und Arbeit auf ein notwendiges Minimum. Da Karten damals wie heute für die Betrachtung und Interpretation durch den Menschen geschaffen wurden, erzielen automatisierte Verfahren, die an die menschliche Wahrnehmung angelehnt sind, die besten Ergebnisse. Die Verarbeitbarkeit, Analysierbarkeit und Vergleichbarkeit historischer Karten wird durch die Bereitstellung editierbarer Vektordaten maßgeblich verbessert und so die tägliche Arbeit von HistorikerInnen, BibliothekarInnen oder StadtforscherInnen unterstützt. Eine Zuweisung semantischer Informationen ermöglicht es Nutzenden beispielsweise nach Schlagwörtern oder Straßennamen zu suchen, Maße von Gebäuden abzuleiten oder deren Genese zu analysieren. Diese Arbeit verdeutlicht die Effizienz eines solchen holistischen Ansatzes und trägt damit zu einer verbesserten Übermittlung und Wahrnehmung geographischer Informationen bei. Durch den Vergleich urbaner Geodaten verschiedener Epochen werden raumzeitliche Veränderungen und Entwicklungen der Menschheitsgeschichte deutlich. 1 Introduction 2 This dissertation investigates historical maps on a large scale with a focus on urban areas. As important trading centers where life pulsates, cities are characterized by long-term changes, shrinkage, and growth. To answer spatial questions about the urban past, the extraction of geometrical as well as of semantic information from large-scale historical maps is crucial for the work of historians, librarians, archaeologists, geographers, and cartographers. Whereas reading these maps is an intuitive process in human perception, their interpretation and analysis are challenging and not trivial. Deriving meanings from historical maps requires the extraction of further information. A large majority of existing historical maps is available solely in paper form. Only a very small amount of these are digitized as raster images and even less have a coordinate system or another spatial assignment. However, by providing digital versions of these maps, the fragile and often unwieldy originals can be preserved and their accessibility substantially increased (Cartography Associates, n.d.-b; Jenny & Hurni, 2011). To make historical maps analyzable in their entirety, a vectorization of their content and an establishment of appropriate databases may be considered as further steps (Knowles, 2008). These approaches have been examined and put into practice for very few and selected map examples and map features so far (cf. chapter 1.2) (Chiang, 2010; Chrysovalantis & Nikolaos, 2020; Gede et al., 2020; Gobbi et al., 2019; Groom et al., 2020; Heitzler & Hurni, 2020; Iosifescu et al., 2016; Peller, 2018; Uhl et al., 2017; Xydas et al., 2022; Zatelli et al., 2019). To address this gap and develop solutions for shortcomings in existing approaches, this thesis aims to bring the work with historical maps one step forward by - enabling and improving the readability, analyzability, and comparability of large-scale historical maps, - making a localization and identification of single objects such as buildings from these maps possible, - improving, combining, and making existing approaches concerning the aforementioned attempts applicable, and - implementing these improvements by a certain degree of automation. With these objectives in mind, the way is opened to reveal and derive new insights from the past which remained hidden previously. The following chapter further specifies the necessity of this thesis based on the current state of the art. 1.2 State of research and research gaps Already at the end of the 1980s, it was advocated to approach the large number and manifold use of maps by automating the extraction process of their valuable information content (Kasturi & Alemany, 1988). Analog paper maps become machine-readable, searchable, and analyzable by extracting and providing their contents in a suitable, e.g., vectorized, data format (Chiang, 2015). This is still often done manually in a laborious way so that an automation of the process is needed (Schlegel, 2019). In this thesis, an all-round workflow for semi-automated extraction of information from large-scale historical maps is presented. A closer look at the literature on this topic reveals a number of challenges, shortcomings, and gaps. In previous studies, there is a lack of preliminary needs assessments to identify actual requirements as well as existing challenges and demands among users of historical maps (Baltsavias, 2004). Instead, authors focus on the development of applicable methods with the aim e.g., to compare the same building between different ages or to detect urban sprawl over time. For this purpose, map content is usually extracted based 1.2 State of research and research gaps 3 on the geometries of map objects (buildings, places, streets, etc.) on the one hand and descriptive labels on the other hand. Together, objects and labels are the most prominent and informative features in largescale historical maps. Therefore, researchers have investigated a variety of approaches to extract this information from historical maps in a semi-automated, timeand labor-saving way. However, an integrated solution is so far lacking. Label extraction The automated detection and recognition of text from digital images have been widely discussed in the scientific literature (Babu et al., 2010; Bhowmik et al., 2018; Chen et al., 2012; Coates et al., 2011; Neumann & Matas, 2013; Nevetha & Baskar, 2015; Weinman et al., 2014; Yao et al., 2012; Ye & Doermann, 2015; Zhou et al., 2017). Identifying text, i.e., labels, from scanned historical maps is particularly challenging due to unique handwritings or nonuniform, noisy, and complex backgrounds. Overlapping map features of similar colors or shapes make an automated and clear distinction of labels even more difficult (Chiang & Knoblock, 2014; L. Li et al., 2000; Milleville et al., 2020; Pezeshk & Tutwiler, 2011; Yu et al., 2016). Previous approaches are often limited to the extraction of specific fonts (Nazari et al., 2016) or uniform sizes (Simon et al., 2014) or orientations (Wang & Yan, 1994) of labels. The process of extracting text is two-staged and composed of an initial detection and a successive recognition part. Text detection aims at the identification of appropriate image regions. Numerous approaches are already suggested for this purpose, for instance, simple color thresholding (Dhar & Chanda, 2006), binarized connected components (Pouderoux et al., 2007; Roy et al., 2007; Wang & Yan, 1994), linear feature extraction (Pezeshk & Tutwiler, 2011), template matching (Budig & van Dijk, 2015; L. Li et al., 2000), or deep learning (Chiang & Knoblock, 2014; Laumer et al., 2020; Weinman et al., 2019). Text recognition is the process of reading character strings from detected text image areas and is usually performed via optical character recognition (OCR). Text detection and recognition approaches can often be found within end-to-end solutions, which are frequently summarized by text extraction (Ye & Doermann, 2015). Very few studies go into further details and consider spatial linkage of extracted labels throughout the map. The extraction of text and objects are frequently regarded separately as standalone processes. However, only a combined processing of map labels and other objects enables the entire extraction of valuable information from historical maps. Object extraction A widespread methodology for the extraction of real-world objects from bitmaps is image segmentation. It divides a raster graphic into meaningless but homogenous regions, so-called segments. Pixel-based segmentation approaches, such as thresholding, maximum likelihood classification, or clustering, solely operate on the basis of color differences between pixels. As a digitized image’s pixels simply represent the color “picked up by the scanner” (Longley et al., 2015, p. 70), pixel-based methods are unsuitable for noisy, complex historical maps with limited spectral information (see example in Figure 1) (Lladós et al., 2001; Zatelli et al., 2019). Object-based image analysis (OBIA), on the contrary, additionally considers e.g., shapes, textures, or spatial contexts of segments and has therefore proven to be more applicable for the extraction of actually existing objects from historical maps. For example, le Riche (2020) also regards differences in textures, while Gobbi et al. (2019) and Zatelli et al. (2019) involve the size and shape besides colors for segmenting historical maps into individual objects. Besides segmentation, OBIA includes an additional classification of segmented image areas into real objects. Thus, plain segments that previously 1 Introduction 4 contained information solely about their topology can be assigned their thematic meanings (Neubert, 2005). Figure 1: Left: Apparently homogeneous background of a historical map; right: close up of highlighted area showing a heterogeneous, complex background structure, which may be caused by the map material’s structure, aging, dirt, or scanning processes Further research on object extraction from historical maps use legend-driven approaches. As described by Lladós et al. (2001), map objects are identified based on their visual signature’s correspondence with elements from a legend. This methodology, however, is ineligible as large-scale historical maps rarely have legends. Instead, customized GIS operations are applied (Chrysovalantis & Nikolaos, 2020; Gede et al., 2020; Gobbi et al., 2019; Iosifescu et al., 2016; le Riche, 2020; Zatelli et al., 2019) and also deep learning techniques become more frequent for object extraction processes (Heitzler & Hurni, 2020; Jiao et al., 2020; Uhl et al., 2017; Xydas et al., 2022; Zhao et al., 2022). The latter, indeed, have a high degree of learning ability, but require a large amount of training data, which is rarely available and therefore often created manually (Gobbi et al., 2019; Lladós et al., 2001; Zhao et al., 2022). Map comparison This thesis addresses the need for a holistic approach to unlock and deduce geographic information, i.e., objects and labels, from large-scale historical maps. With the help of an integrated extraction process, new insights of the past can be gained. Objects can be identified based on the labels’ semantic meaning, informative databases may be generated, and quantitative evaluations and spatial analyses become feasible (Zhao et al., 2022). Direct comparisons with other geodata allow for identification and visualization of changes over time within cityscapes. However, it is still insufficiently explored how identical real-world objects may be identified and matched between different maps in an automated way. In the field of road management, location referencing methods are applied to unambiguously identify objects from different maps (International Organization for Standardization, 2022; Kenley et al., 2019). This may be intended to e.g., visualize long-term developments of individual objects or to define suitable control points for georeferencing purposes. Existing approaches match identical geometries from different maps based on their colors (Stefanidis et al., 2002), shapes (Xavier et al., 2016), attribute values (Frank & Ester, 2006), or spatial relations (J. O. Kim et al., 2010; Samal et al., 2004; Sun et al., 2020; Tang et al., 2008) but seem to be impractical for complex, monochrome historical maps lacking these characteristics or any further information and georeferencing. Regardless of extracted map features, Balletti and Guerra (2009) determine the general correspondence between different historical maps, but their color-based approach only works for maps of the same style or series. Similarly, El-Hussainy et al. (2011) measure gray value differences to determine the similarity between historical and more recent maps. To analyze long-term changes of forest areas, Loran et al. (2018) compare small-scale maps from several ages based on, for instance, their metadata. Ory et al. (2017) suggest 1.2 State of research and research gaps 5 an alternative approach for comparing historical with current maps by creating multiple in-between representations (“cartographic continuum”). By interpolating colors, contour lines, and shading of map features, they were able to merge and blend cartographic styles from different ages. Common limitations Previous studies on information extraction from historical maps and their comparison to others often require an existing georeferencing (Chrysovalantis & Nikolaos, 2020; El-Hussainy et al., 2011; Gede et al., 2020; Gobbi et al., 2019; Iosifescu et al., 2016; le Riche, 2020; Loran et al., 2018; Piechl, 2020). Moreover, many approaches are restricted to particular map styles and are therefore non-transferable. Black-and-white cadastral plans, colorful or high-contrast maps, distinct geometries, small-scale maps with larger and more homogeneous areas, or discrete map features have already been examined. But, in fact, blurring, overlaps, and low color gradations rather reflect reality and are challenging for researchers in this domain. Historical maps are subject to large heterogeneities in style, condition, and year of creation. Adapting and optimizing information extraction processes is therefore essential. To minimize human subjectivity in this context, manual intervention as well as preand post-processing are to be reduced. While authors agree that full automation of extracting information from historical maps is impracticable (cf. Bucha et al., 2005; Budig, 2016; le Riche, 2020; Simon et al., 2014; Stefanidis et al., 2002), end-toend semi-automated approaches can be used to diminish time, mis-interpretations, and other humaninduced errors. There is no consistent, universal workaround, which is designed “for practical use” (Baltsavias, 2004, p. 131) and combines all necessary processes for extracting information from large-scale historical maps to date. Single processes within a holistic system may even benefit from each other and therefore maximize the degree of automatization. With the development of transferable, holistic solutions, new geographic information can be generated in a GIS-ready format and used for the creation of databases, for multitemporal analyses, and change detections throughout human history. Considering the mentioned shortcomings, the following major issues are pursued in this dissertation with the aim to deduce urban developments from large-scale historical maps: - Accelerating and simplifying existing procedures for information extraction by semi-automation while minimizing human intervention, - developing a comprehensive end-to-end workflow from information extraction to spatial assignment, - taking advantage of object-based image analysis, which originates from the field of remote sensing, with regard to object extraction from large-scale historical maps, - combining objects with their semantic meanings and, potentially, further derivable and related information, - improving the process of linking maps from different ages to enable a more intuitive comparison, and - minimizing common restrictions concerning specific map styles for a workflow to be as transferable and universal as possible. 1 Introduction 6 1.3 Structure of the thesis This dissertation has a cumulative form and consists of a general framing text followed by three published scientific papers, which can be found in Appendices A to C. The framing text gives an introductory overview of the general research objectives. Stated research questions bring the publications into a thematic context and summarize the applied methodologies and major results. After a short summary of each publication, chapter 2 provides basic information on historical maps in general, being the superordinate objects of research of this thesis. Methodological fundamentals concerning information extraction from historical maps are also given here. Chapter 3 introduces the main research objectives and questions addressed by this thesis, while answers to these are given in chapter 4. This part provides the centerpiece of this thesis by summarizing major results and applied methodologies. The concluding chapter 5 discusses the key findings and gives a final outlook on future developments and perspectives this dissertation can provide. 1.4 Publication overview All three papers published within the framework of this dissertation aim at the simplified identification and extraction of information from large-scale historical maps. A summary of each of the publications is given in the following. Figure 2 illustrates their major findings. Figure 2: Results of the three published papers: a) Large-scale historical map of the city of Hamburg with an intuitive semiology according to a conducted user study (Paper I); b) Historical map spatially transformed to a dataset with current road names based on a fuzzy toponym matching (Paper II); c) Extracted and vectorized map objects (Paper III) 1.4.1 Paper I: Requirements analysis Investigating historical maps is a complex and time-intensive task, in which users often struggle due to visual information overload (Christophe et al., 2016). However, present needs and requirements regarding the usage of historical maps have not been examined to date. Therefore, a survey3 among users (e.g., historians, librarians, archivists, publishers, and urban researchers), who deal with historical maps in their daily working routine, was conducted as part of Schlegel (2019) (see Appendix A). The focus of this user study is on the human visual perception of large-scale historical maps. Major challenges in the investigation and comparison of historical maps could be identified. Besides distortions and great visual variations within maps, users stated the general lack of suitable processing tools as a reason of struggling in working with historical maps. When comparing historical maps with other historical or current counterparts, users regularly apply individualized GIS operations or even manual procedures with analog paper maps. In doing so, their research primarily focuses on the comparison of certain objects. They examine the temporal 3 https://github.com/IngaSchl/User-Study-Historical-Maps/blob/main/User-Study.pdf 1.4 Publication overview 7 change of buildings and roads in particular, which commonly depict the major urban structure in largescale historical maps. Apart from the described problem analysis, this publication further examines human intuition regarding the identification and differentiation of map objects from exemplary large-scale historical as well as current maps. It was found that semiological characteristics (cf. chapter 2.1.3) such as colors, contours, textures, symbols, and labels significantly contribute to these processes. Especially distinct color differences and high contrasts make the recognition and differentiation of map features easier (see example in Figure 2 a)). In the exploration of commonly monochrome or colorless historical maps, intuitive textures and labels are regarded as suitable for such tasks. This initial publication therefore provides a knowledge base for further research on the extraction of information from large-scale historical maps as well as their comparison to other historical or more recent counterparts. These matters are addressed within the following two papers. 1.4.2 Paper II: Label extraction To deduce long-term urban developments, large-scale historical maps should be analyzable and comparable. An essential first step towards this objective is to render their information content machinereadable. As one of the most prominent sources of information, map labels reveal the names of streets, places, and buildings. Therefore, an automated end-to-end workflow combining tools to detect, recognize, and match labels from large-scale historical maps was developed in the context of Schlegel (2021) (see Appendix B). As authors often struggle with differentiating text from other map features mainly due to similar colors, this study suggests the application of a universal machine-learning tool called Strabo (cf. Z. Li et al., 2018). Strabo enables the detection of textual image areas not only based on color but also on text size and neighborhood relations. Different sizes, orientations, and curvatures of text and even overlapping labels could thereby be identified from an exemplary, slightly colored historical map (cf. chapter 4.1). To further read character strings from detected text image areas, the free and open-source tool Tesseract (cf. Tesseract OCR, 2019) was used, which is well-known for optical character recognition. Both detection and recognition rate could be increased through image enhancements and supplementary improvements resulting in a f-score – the harmonic mean of precision and recall – from formerly 0.58 to up to 0.77. A final comparison of recognized character strings to current names of streets and places within the considered map section was conducted to verify the historical labels. With the suggested solution, a spatial assignment between historical and current map features covering the same area was enabled (see Figure 2 b)). By using concurrent toponyms (according to the definition by Imhof, 1972) for the definition of reference points, a rough georeferencing of the historical map could be performed, which further facilitates the comparison between maps from different ages. The presented solution considerably reduces manual preand post-processing and is largely applicable to other, comparable large-scale historical maps. In the long term, the solution aims at retrieving relevant information to be further managed and stored on the one hand and at temporarily eliminating text from historical maps to facilitate the identification of further objects on the other hand. 1.4.3 Paper III: Object extraction One further essential step towards machine-readability of historical maps is the digital reproduction of map features representing physically existing objects such as buildings, streets, or water areas. In Schlegel 1 Introduction 8 (2023) (see Appendix C), a holistic workflow to semi-automatically extract objects from large-scale historical maps is presented. The approach consists of 1) an initial elimination of labels, 2) an extraction and vectorization of geometries, primarily of buildings and water areas, 3) a linking, as well as 4) a rough spatial transformation of corresponding current geodata to enable a spatial assignability of the historical map features. By using bounding boxes containing textual image areas as an outcome from Paper II, an improved basis for the following steps could be created: Unlike other studies, which ignore or treat labels as disturbing elements in the course of object extraction, the presented procedure clearly separates between labels and touching or overlapping map objects of the same color. Therefore, the original shapes of e.g., buildings remain unaltered and can be detected in their entirety in the following step of object extraction. This was conducted via object-based image analysis, which differentiates, extracts, and classifies objects based on graphical variations similar to human perception. By regarding not only colors but also textures, shapes, or spatial contexts, OBIA is particularly applicable for monochrome or colorless, hand-drawn, as well as aged and smudged historical maps. To spatially assign the resulting, vectorized objects (see Figure 2 c)) from the large-scale historical map to more recent counterparts, shapes of churches and municipal buildings were matched as these barely change over time. Centroids of those geometries having a great shape similarity served as control points in the course of a following semi-automated affine transformation. Thereby, a spatial rectification of current geometries was performed. Resulting average deviations of 34 m in reality were deemed satisfactory. The workflow demonstrated in Paper III not only supersedes non-transferable, error-prone, as well as timeand labor-intensive manual attempts but also provides searchable and comparable data to derive new knowledge from large-scale historical maps. The resulting vector data may be further evaluated and analyzed in GIS, for instance. Related databases can be generated by using gained and other derivable information and thereby allow for the investigation of urban morphological developments. 9 2 Historical maps To provide a deeper understanding of historical maps, this chapter first specifies background information concerning the general object of research. In the second part, common methodological principles for working with historical maps and their application in this thesis are described. 2.1 Object of research 2.1.1 Genesis The first maps recorded in human history were engraved in clay and can be dated back to antiquity (Dodge et al., 2011). At that time, maps served to show religious worldviews or for orientation purposes. For example, locations that provided drinkable water or fertile land could be identified. With accuracy increasing over the course of time, also locations, sizes, and borders have been described by maps (Crom, 2013; Goss, 1994). Claudius Ptolemy, an astronomer in the second century A.D., provided a formative base in cartography by developing a mathematical global system consisting of latitude and longitude. He thereby pioneered the two-dimensional representation of the three-dimensional globe (Livieratos, 2006; Thompson, 2017). Ptolemy maps, which rely on his latitude/longitude model, dominated next to mappae mundi and portolan charts in the Middle Ages. Mappae mundi are characterized by detailed and colorful but unscientific maps portraying Christian beliefs while portolan charts were created for navigation purposes on sea voyages (Goss, 1994). With Gerardus Mercator and his eponymous, conformal map projection, accurate navigation became possible for seafarers from the 16th century on. Further innovative developments between the 16th and 18th century were achieved with even more precise calculations of distances, areas, the meridian arc, and with triangulation (Crom, 2013; Goss, 1994; Thompson, 2017). With an increase in the scale of maps, also city maps were produced by triangulating crossroads, church steeples, and other distinctive buildings. Mathematical and technical developments during the 19th century, such as the theodolite, entailed a novel precision in the fields of surveying and cartography (Andrews, 2009; Medyńska-Gulij & Żuchowski, 2018; Thompson, 2017). Today, a large number of replicas of these maps – which were originally engraved, for instance, in wax, steel, or copper plates or drawn on paper with ink and watercolors – are available in map archives and libraries (Robinson et al., 1995). Digital scans make these valuable sources from the past accessible to the public. 2.1.2 Terminology Historical map is a comprehensive term. A great number of definitions exist concerning the temporal classification of historical maps and even the term itself is much discussed. While “early maps” (Boutoura & Livieratos, 2006; Höhn et al., 2013) or “old maps” (Gede et al., 2020; Iwanowski & Kozak, 2012; Simon et al., 2014; van Dijk & Schommer, 2016) may be misinterpreted as outdated maps from the recent past, “ancient maps” (Drapeau et al., 2017) are rather associated with prehistoric instances. “History maps” are defined as thematical maps showing bygone states at the time of creation by Crom and Heinz (2016). This thesis is based on the well-known term historical maps, which, by the majority of relevant literature, is seen as a simplified representation of previous geographic structures and conditions that was produced in the past (based on Hake et al., 2002). The definition of this past period of time is controversial. As a result of the user study presented in Paper I, which was conducted among 31 German experts (cartographers, historians, librarians, urban researchers, etc.), a rough temporal boundary between 2 Historical maps 10 historical and current maps can be drawn around 1850, the approximate starting time of industrialization in Germany. For this thesis, maps produced before and around this turning point are considered as historical maps. In this context, historical maps are further differentiated by means of their scale – the linear reduction ratio of the map toward reality (Hake et al., 2002). While small-scale maps show large geographic areas, large-scale maps (>1:20,000) portray smaller areas in more detail such as cityscapes or neighborhoods including e.g., single buildings and streets. This thesis focuses on large-scale historical maps illustrating urban landscapes and originating from the above-mentioned period of time. 2.1.3 Semiology Human perception (cf. chapter 2.2.1) interprets the visual appearance or style of a map based on similarities and differences between map elements. Users perceive a map’s visual signature due to different graphical signs and recognize known structures and criteria. The visual image of a map is affected by the graphical representation as well as the geographic context and spatial relations of map objects. Generally, map style intends to correctly represent the content of a mapped region and to reveal a map’s meaning and is therefore primarily characterized by its purpose. The style of a map can be of social, emotional, or aesthetic nature and allows to draw conclusions about a map’s origin, such as its author and period of production, and target group (Beconytè, 2011; Crom, 2013; Ory et al., 2013, 2015, 2017). So-called visual variables build the foundation of any image. In maps, they represent geographic features and processes by encoding cartographic information and thereby compose a map’s overall design. Based on the viewer’s experience, visual variables can improve the perception and interpretation of a map (Schlichtmann, 2017). As cartographic communication is strengthened by using visual variables, users are able to immediately analyze the content of a map (Beconytè, 2011; Ory et al., 2015). Humans perceptually see visual variables, which then are “processed by the eye-brain system” (Roth, 2017, p. 5). For the first time, visual variables were defined in Jacques Bertin’s Sémiologie Graphique from 1967. His work specifies rules for the graphic representation of different information and is still considered a theoretical basis in cartography and information visualization. According to Bertin (1967), the sum of all visual variables – specifically the size, shape, location, orientation, texture, as well as color hue and value – can be defined as semiology, which forms the visual appearance or style of a map (Christophe, 2012). Often, labeling is considered as an additional element of a map’s semiology. Both a whole map and individual map objects may have a semiology (Dodge et al., 2011; Roth, 2017). Regarding historical maps, style is commonly neutral in terms of emotions. Unlike e.g., a precise scaling or accuracy, a user’s ability to orient themself with the help of the map was a key factor for a map’s quality in previous times. Maps were designed for intuitive readability and legends not considered as necessary elements by mapmakers (Beconytè, 2011; Crom, 2013; Medyńska-Gulij & Żuchowski, 2018). Being considered the foremost element in the representation of spatial information within maps, color can be easily combined with other visual variables. While similar colors point to similar or related objects, color contrast is used to improve the perception of hierarchies between areal map objects (Herold, 2018; Larcher & Piovan, 2018). Such variations in visual variables help to differentiate map contents in a qualitative and quantitative way. Cartographic color conventions, such as blue for waters, red for buildings, or yellow to brown for roads, were introduced from the 15th century on and are still in use today. However, with regard to predominantly colorless historical maps, their semiology is characterized less by colors but rather by textures, line widths and shadows, symbols, and labels (Andrews, 2009; Goss, 1994; Medyńska-Gulij & Żuchowski, 2018; Roth, 2017). 2.2 Methodological fundamentals 11 When processing historical maps, especially in a (semi-)automated way, their unique styles must be taken into account. Dependent on the technique used by the cartographer or draftsperson and their personal signature, not only the visual appearance of historical maps, but also scale, accuracy, and level of detail vary greatly (Beconytè, 2011; Losang, 2015; Ruggles, 1982). 2.2 Methodological fundamentals This chapter gives an introduction into the principles which were applied throughout the three papers published in the framework of this thesis. They all describe common proceedings to be taken into account while working with large-scale historical maps and aim at the extraction of information from these. The tasks are explained in the subsequent sections in the following order (see also chapter 1.4): 1) In Paper I, it was examined how humans perceive and interpret large-scale historical maps. 2) Derived from human processes of perception and interpretation, Papers II and III address the semi-automated extraction of labels and objects, respectively. 3) Recommendations on how this extracted information can be used to spatially transform historical and current maps without major manual effort are described in Papers II and III. 4) As a result of spatial transformation, a semi-automated methodology for comparing large-scale historical with current maps was developed in the course of Paper III. 2.2.1 Human perception The human eye senses reflected light, which gets transmitted to the brain. There, cognitive abilities allow us to recognize and differentiate between single objects, in the case of map interpretation buildings, roads, water or green areas, etc. Visual detection of individual objects is based on e.g., tones and saturations of colors, patterns, shapes and sizes, as well as absolute and relative locations. Human perception is founded, on the one hand, upon graphical variations, artifacts, and deviations and, on the other hand, upon experience, to distinguish between objects and to further connect them based on e.g., neighborhood relations (Blaschke et al., 2014; Dent et al., 2009; Herold, 2018; Neubert, 2005; Ory et al., 2015). In accordance with the Gestalt laws of perceptual organization by Wertheimer (1923) and Goldstein (2008), humans tend to perceive simple and concise structures (law of simplicity) and see similar elements belonging together (law of similarity) (Herold, 2018). OBIA, for instance, makes use of this object-driven approach by segmenting and classifying an image into single, meaningful objects. As historical maps in particular were created to be read by humans, OBIA is a promising solution for automating human perception processes regarding object extraction (Ablameyko et al., 2001). Labels and legends can further help to identify and assign objects to classes with the latter rarely being available in large-scale historical maps (Medyńska-Gulij & Żuchowski, 2018). 2.2.2 Object and label extraction With scanning an analog paper map, a digital version of a historical map can be provided. Many online platforms from libraries, scientific institutions, or private archives supply historical maps as bitmaps to make them widely accessible to the public (Novak & Ostash, 2022). Often, digitizing is used as a synonym for scanning historical maps, but the term is ambiguous as it may also describe a latter vectorization of map objects. Woods et al. (2016) give an overview of scanning methodologies for historical maps. With the availability of high-resolution scans, an appropriate basis for the computerized extraction of information from historical maps is established. The extraction of labels and objects from large-scale 19 4 Methodologies and key results In this chapter, the main scientific findings and methodological implementations of this dissertation are presented. After introducing the investigated data basis, answers to the research questions defined above are given. 4.1 Investigated data basis The fundamental data basis used for demonstration purposes of the workflows developed in the framework of this dissertation is an exemplary large-scale historical map. Originally produced as a steel engraving in the middle of the 19th century on a scale of approximately 1:11,000, the map in Figure 5 shows the city center of Hamburg. The map section illustrates the most relevant elements of the original map, i.e., buildings, streets, water areas, and labels. This sample was already part of the user study in the context of Paper I and was further processed within Papers II and III. A digital scan of the map with a resolution of 300 ppi is provided by the Harvard Map Collection et al. (n.d.-a). Figure 5: Section of the large-scale historical map used as an exemplary object of study in this thesis (Harvard Map Collection et al., n.d.-a) Additional maps and geodata depicting the same spatial area were used in the further course of this dissertation. In Papers II and III, additional historical maps primarily served to verify developed workflow processes, whereas datasets containing current streets or buildings were used for spatial transformation purposes. Detailed sections of current maps were used to investigate human perception of semiology in Paper I. More details on the input data can be found in the three publications that are reprinted in the appendix. 4 Methodologies and key results 20 4.2 Answers to research questions In the following, concrete answers to the research questions formulated in chapter 3.2 are provided. Thereby, an overview of applied methods and obtained results within this thesis is given. More detailed discussions can be found in Appendices A to C as part of the publications. Further descriptions and results from single workflow steps can also be found in the publications as well as the corresponding GitHub repositories7, 8. 4.2.1 Required information from large-scale historical maps RQ1.1 What are common subjects in the exploration of large-scale historical maps? What do users intend to investigate when comparing historical with current maps? Which questions do the users pursue? Methodology: In the course of Paper I, a pen-and-paper survey3 was conducted with 31 selected historians, librarians, archivists, and urban researchers in order to answer this research question. Results: More than half of the study participants work with historical maps several times per week or at least once a month. Besides archiving, digitizing, and cataloging map stocks, comparing historical maps with current counterparts is a key task among users. Statistical evaluations of the user study revealed a major focus on the exploration of historical maps: the investigation of long-term changes of individual map objects, primarily buildings and roads. These changes may be of shaping or semantic nature. For instance, buildings may have been extended or street names might have changed. Also, these map objects are often examined for simple existence or absence over the years. Such transformations affect long-term urban developments, which often account for the research of large-scale historical maps. According to related literature, historical structures and conditions can frequently not be deduced from other primary sources like writings describing the past, but only become visible from historical maps (Chiang et al., 2020; Harley, 1987). Users even verify historical records and other cartographic material by means of historical maps. RQ1.2 How can textual information from historical maps, particularly in the form of labels, be identified and extracted? Methodology: Based on a literature research on the extraction of labels from historical maps, common tools for text detection and recognition were chosen. The application of minor image enhancement techniques as well as the additional usage of various Python libraries helped to considerably improve former processes. A comprehensive workflow for label extraction from large-scale historical maps could thereby be developed. Results: Extracting semantic information from a historical map allows for searchability and analyzability of places in the course of time (Chiang et al., 2020). In the recent past, combining text detection and recognition methods has successfully proven to be able to extract machine-readable textual content from historical maps. The application of deep learning-based OCR for text recognition is, on the one hand, adequate in terms of labor input, computing time, and recognition rate, but, on the other hand, insufficient as backgrounds of historical maps are not generally plain white or homogenous. A preceding detection, which identifies textual areas within the input map, is therefore required (Nazari et al., 2016; Ye & Doermann, 2015). For this purpose, Z. Li et al. (2018) provide the ready-to-use tool Strabo. This deep learning text detection system identifies text pixels based on “cartographic labeling principles” 7 https://github.com/IngaSchl/Label-Extraction 8 https://github.com/IngaSchl/Object-Extraction 4.2 Answers to research questions 21 (Chiang et al., 2016, p. 29) regarding e.g., color, size, and contextual criteria of potential characters. Preceding image enhancements (e.g., linear contrast stretching or global histogram equalization) and rotations of the input map can be helpful to increase the quantity of detectable text image areas. As demonstrated in Paper II, a horizontal alignment of these detected textual image areas also multiplies the number of true positives within subsequent text recognition processes. Detected map labels can easily be transformed into machine-readable character strings by using an OCR engine like Tesseract (cf. Tesseract OCR, 2019). A final fuzzy matching (e.g., via Levenshtein Distance, as suggested by Yu et al., 2016) of these strings allows a linking of historical toponyms to today’s similar or even unchanged names of e.g., roads and places (Milleville et al., 2022). By combining tools for text detection and recognition with the mentioned enhancements, the extraction of labels from large-scale historical maps can be considerably improved. A detailed schematic of the described workflow, precise evaluations, as well as an explanation of inappropriate approaches can be found in Paper II. RQ1.3 Which non-textual information can be extracted from large-scale historical maps and how? Besides labels, this dissertation aims at extracting further objects from large-scale historical maps. Buildings, roads, as well as water and green areas are typical elements. However, the extraction of individual objects is rarely straightforward. As can be seen from Figure 5, discrete roads, for instance, have no closed geometries. Instead, the map resembles a modern figure-ground diagram, which solely represents building footprints while omitting geometries of streets and water or green areas (Mueller-Haagen et al., 2014). As per the results of the user survey conducted in the context of Paper I, buildings are one of the most studied objects in large-scale historical maps. Therefore, this thesis predominantly handles the extraction of those. Methodology: Based on their semiological components such as color, texture, and shape, but also e.g., neighborly relations, objects from large-scale historical maps can be extracted by applying OBIA. A brief outline of the methodology applied in Paper III is shown in Figure 6. Detailed process chains, which can be directly implemented in eCognition, are provided in a related repository8. Figure 6: Simplified scheme of extracting geometries of (public) buildings and water areas from a large-scale historical map in eCognition (GLCM = gray-level co-occurrence matrix; GLDV = gray-level difference vector) Results: As shown in Figure 6, the historical map described in chapter 4.1 is used as an exemplary input. Splitting this bitmap into its individual RGB and HSV channels was done to emphasize single objects. An initial global thresholding was applied to separate dark foreground objects, such as buildings and labels, from brighter streets, places, and water areas in the background. After classifying labels – which were already identified in Paper II – also as map background, shapes of buildings could be concluded from the map. A subsequent edge detector can define precise object contours by means of a gray-value gradient. 4 Methodologies and key results 22 The identification of building geometries in the map could be further improved based on their hatched texture. A two-dimensional gray-level co-occurrence matrix (GLCM) considers the vertical invariance of pixel combinations. By quantifying the combination of gray value pixels adjacent to each other, a GLCM identifies repeating, e.g., hatched, patterns (Chaves, 2021; Trimble Inc., 2021). In Figure 7, the general principle of a GLCM is demonstrated. If an underlying image is homogeneous, the largest values will be found in the GLCM’s diagonal as two adjacent pixels are of the same gray value. Figure 7 a) shows the opposite case, a heterogeneous, i.e., hatched texture of a typical building within the examined map. Figure 7: Gray level co-occurrence matrix (GLCM) of an exemplary buildings’ close-up (a)). Based on a simplified tricolored version (b)) of the input image, each pixel’s relationship to its neighbor below (c)) is added up in the GLCM. As a last step, the resulting GLCM (d)) is normalized (e)) to get the probability of each gray value pair following their vertical relationship As can be seen from Figure 6, another thresholding operation applied on the map’s HSV value or brightness channel helped to differentiate darker public buildings. Moreover, a gray-level difference vector (GLDV), which is another textural heterogeneity measure for the identification of local variations, proved useful for extracting the textured and colorized water areas. After extracting the mentioned map objects, a raster-to-vector conversion, e.g., by GDAL or other GIS methodologies, allows a straightforward linkage of these objects with toponyms extracted within the framework of Paper II. Thereby, related databases may be generated. OBIA has only been marginally used in conjunction with historical maps to date. In principle, OBIA is applied in the field of remote sensing. Results from Paper III demonstrate, however, the usefulness of object-based image analysis in the object extraction from large-scale historical maps. Alternative approaches, such as simple histogram thresholding, color space clustering, or artificial neural networks proved unsuitable as spatial relations between objects are neglected or a large amount of training data and time are needed (Gobbi et al., 2019). If applicable, an additional extraction of map symbols would be recommended to derive further relationships of map objects. However, symbol extraction is rather an issue in middleor small-scale maps (see Garcia-Molsosa et al., 2021; Groom et al., 2020; Szendrei et al., 2011). RQ1.4 In which form should large-scale historical maps be made available to make them comparable to current equivalents? Are there any recommendations concerning technical specifications such as the data format or resolution? Methodology: To give an appropriate answer to this research question, a literature research was conducted. Results: Scans of large-scale historical maps ideally have a scanning resolution of 300 ppi for information extraction purposes (Chiang, 2010; Chiang et al., 2014; El-Hussainy et al., 2011; Peller, 2018). In literature, higher pixel density values of up to 600 ppi are often recommended, but interfering artifacts from paper folds, smudges, or discolorations may then be overemphasized (Groom et al., 2020; Iosifescu et al., 2016). Moreover, scans of historical maps are frequently provided at a resolution of approximately 300 ppi. This value represents a compromise between the visual degree of detail and required storage 4.2 Answers to research questions 23 space. Regarding the file size, an image’s bit depth is an additional influencing factor. Often, 8-bit images with 256 possible pixel values per color channel are used for the information extraction from maps (Ablameyko et al., 2002; Ekim et al., 2021; Gobbi et al., 2019; Zatelli et al., 2019). As original pixel information cannot be maintained by, for instance, compressed JPEG images, the usage of lossless TIFF data is advisable (Gede et al., 2020). For a proof of concept of the workflows demonstrated in this thesis, a representative map section (see Figure 5) covering all relevant map elements such as buildings, roads, and water areas proved to be adequate. Such a cropped version of the original saves computing time and effort. Georeferencing, in this case, is rather detrimental as it may falsify or even distort a map’s original content. Instead, a spatial transformation should be performed as a final step of the whole extraction workflow. Compared to the original scan in the form of a raster image consisting of single pixels, vectorized data offers many advantages and considerably facilitates the comparability of historical map content. Additional information on map objects like extracted names of streets or buildings can be recorded as attribute values in a database. Such in-depth information considerably simplifies the searchability, processing, analysis, and comparability of historical maps. Besides, vector data needs less memory and allows further processing steps like rectification or smoothing of its outlines or the application of an alternative semiology. A final raster-to-vector conversion of extracted objects is therefore highly recommended “to produce a higherlevel approximation to the geographic elements” (Liu et al., 2019, para. 5). RQ1.5 Which semiological characteristics are most suitable to intuitively identify and differentiate contents – such as objects and labels – of large-scale historical and current maps? Methodology: This issue has been investigated within Paper I based on findings of the user study conducted in this context. Here, the suitability of different visual variables is elaborated in more detail by involving insights also from additional literature. Results: The user study revealed that the identification and differentiation of objects by humans is not based on their familiarity with a map but rather on their awareness of the semiology of objects. Typical associations are a blue coloring for bodies of water or green for forest areas (Medyńska-Gulij & Żuchowski, 2018). However, as colors are rare in historical maps, contrasts, textures, and labels are the most important visual variables contributing to an intuitive identification of single map elements and different object classes. In Paper I, it was found that texture is the best perceivable semiological characteristic. This applies, in particular, to the recognition and differentiation of buildings from large-scale historical maps. Regarding maps from present times, also line widths are crucial to distinguish objects such as buildings, roads, or water bodies. This is especially true for low-contrast or visually overloaded maps. Also, the automated readability of map content is impaired with limited spectral information in monochrome maps (Herrault et al., 2013). Color, in principle, is the most obvious visual component for the automatic identification of map objects as information can be directly deduced from single pixel values (Chiang et al., 2020; Thenkabail, 2015). A wide range of brightness generally supports the differentiability of object classes. Objects belonging to one class should have consistent colors and patterns. This is, however, rarely the case for historical maps as hand-drawn textures prevail. Also, interrupted contours and thin lines as well as particularly overlapping map elements of uniform colors and stroke widths significantly impede a (semi-)automated extraction of objects or labels (Gladstone et al., 2012; Iosifescu et al., 2016). 4 Methodologies and key results 24 RQ1.6 To which extent can human perception serve as a basis for the development of a semi-automated workflow improving the comparison between large-scale historical and current maps? Methodology: To give an appropriate answer to this research question, a literature research was conducted. Further experience concerning the applicability of human perception in terms of semi-automated processes could be gained during the development stages. Results: As described in chapter 2.2.1, humans intuitively identify map objects by means of similarities and differences in color, size, texture, shape, spatial context, etc. OBIA adopts this principle by segmenting and classifying images based on multiple visual variables simultaneously. In accordance with human perception, object-based image analysis does not follow a mere pixel-wise approach. Instead, the examined image or map is split into uniform objects based on e.g., spectral and shaping homogeneities (Gladstone et al., 2012; Hussain et al., 2013). Also, neighborhood relations play an important role in manually assigning and linking information and should therefore also be considered within (semi-)automated approaches. If objects or labels are extracted from a historical map by a machine, a contemplation of its HSV or CIELab color space may be helpful as this rather corresponds to human perception compared to the use of the RGB color space (Dodge et al., 2011; Ostafin et al., 2017). In Paper III, the HSV value channel, which represents the brightness of colors, was used repeatedly to differentiate between object classes in a semiautomated way (see Figure 6). By incorporating cognitive strategies within a semi-automatic workflow, the comparison between historical and current maps can be facilitated to a certain degree. With automation, constant eye movements alternating between saccades and fixations as well as elaborate matching procedures between heterogeneous semiology are no longer necessary. 4.2.2 Automation of the information extraction workflow RQ2.1 What does automation mean? Many authors develop and apply semi-automated approaches but rarely define this term. Which degree of automation defines automated, semi-automated, or non-automated approaches? Methodology: To give an appropriate answer to this research question, a literature research was conducted. Several findings could be verified and extended by results presented in Papers II and III. With the help of multiple data input scenarios, the developed workflows’ degree of applicability, amount of needed manual interaction, and saving of running time could be appraised. These factors allow an approximate assessment of the automation level. Results: Automated processes contrast with manual, non-automated techniques, which inevitably require human efforts for the performance of a task. Automation allows to reduce expenditure of time and labor as a particular workflow can be applied to various inputs and produces comparable results, for example, concerning extraction accuracies (Gobbi et al., 2019; Laycock et al., 2011; Liu et al., 2019). Authors occasionally differentiate between fully automated and semi-automated approaches (Milleville et al., 2022; Uhl et al., 2017). However, state-of-the-art technological facilities such as deep learning cannot be seen as fully automated because training data has to be generated in a supervised way. By the current state of scientific knowledge, an automation of information extraction from historical maps to a degree of 100 % is impossible (Bucha et al., 2005; Simon et al., 2014). This is also due to the large number of historical maps having a high degree of information, complexity, and visual variety (N. W. Kim et al., 4.2 Answers to research questions 25 2014). As maps, especially in the past, have usually been “produced to be read by humans” (Ablameyko et al., 2001, p. 195), it is obvious that graphical variations and artifacts could be identified differently by humans than by machines. Furthermore, in the semi-automated process of information extraction from historical maps, manual interaction is still necessary for the initial problem definition, the choice of data, processing methods, and sources of knowledge, as well as for the final verification of the workflow results (Baltsavias, 2004; Herold, 2018). To bridge the gap between user interaction and automation capacity, semi-automated methods are used to improve the quality and efficiency of a former manual workflow (Baltsavias, 2004). Especially if a large number of maps are to be processed in a similar way, manual efforts can be reduced by a certain degree of automation. Contrary to that, automated approaches perform worse with heterogeneous input maps and diverse cartographic styles, as results from Paper III show (Chiang et al., 2016, 2020). Preventive or corrective user interaction is often needed for semi-automated information extraction but should be avoided as much as possible in favor of automation (Baltsavias, 2004). Pre-processing attempts, such as filtering or morphological operations, require thorough consideration upfront as well as individual solutions and usually falsify the original input. Well-known and readily available tools, e.g., Python libraries, however, can be used for post-processing to optimize results from information extraction. The semi-automated workflow for object extraction demonstrated in Paper III can be applied to other large-scale historical maps with minor changes if they meet the following requirements. Maps should have similar or less complexity and a similar scanning resolution and bit depth as the map described in chapter 4.1. In Paper II, additional adjustments to improve the extraction of labels were automated for a whole map sheet (cf. chapter 3.3 in Paper II). Hence, time and effort can be reduced significantly. However, the individual influence of the developer on such semi-automated methods cannot be neglected, which is strongly dependent on their subjective choice of applied methods, thresholds, and parameters. In this context, it is unavoidable to implement minor manual improvements in order to achieve satisfactory results. RQ2.2 To which extent can user interaction be reduced to a necessary minimum? Can prevalent manual procedures, which are timeand labor-intensive, be accelerated or even substituted? Methodology: The processes applied in Papers II and III were evaluated regarding their automation capacity. A tabular juxtaposition of automated and manual proceedings implemented within the workflows of this thesis outlines the possible degree of automation. Results: To minimize human intervention in the process of information extraction from historical maps, additional preliminary work such as the application of filters or other image enhancements should be avoided. The development of preferably universal approaches contributes to the automation and, thereby, the substitution of manual processes. Recurring problems can be solved in a faster and consistent way with the help of (semi-)automated strategies. Table 1 juxtaposes the automated and manual processes being applied or developed in this thesis for the purpose of information extraction from large-scale historical maps. The application of various Python libraries (e.g., NumPy, Rasterio, Shapely), GIS operations, and other software tools contributed to the automation of processes. Nevertheless, single procedures, such as the determination of usable algorithms and functions, parameters, and thresholds, still require human intervention. Additionally, a precise ground truth needs to be generated manually in most cases for a final quality analysis of label and object extraction results. However, apart from the subjective choice of OBIA algorithms, Table 1 shows that these manual processes are rather of minor impact and well-known from other applications like image processing or data 4 Methodologies and key results 26 evaluation. Some manual steps need to be defined only once during development phase. Established tools, such as Strabo or Tesseract, should always be favored. Thus, user interaction could be minimized to a certain degree within the information extraction from historical maps. Table 1: Automated and manual processes needed, used, and developed within this thesis for the purpose of semi-automated information extraction from large-scale historical maps Automated processes Manual processes Label extraction (cf. Paper II) - Multiple rotations of input image to increase the number of detectable textual areas - Merging bounding boxes covering identical textual areas - Rotation of textual image areas to improve the data base for text recognition - Measuring string similarity between recognized text and additional dataset - Affine transformation of historical map - Definition of thresholds and conditions for merging of bounding boxes - Definition of language in Tesseract OCR - Choice of appropriate current dataset for string similarity - Evaluation of results compared with a ground truth (e.g., recall, precision, and fscore) Object extraction (cf. Paper III) - Execution of OBIA rule set in eCognition - Raster-to-vector conversion - Definition of control points for spatial transformation based on shape similarity - Affine transformation of current dataset - Definition of thresholds and conditions for the differentiation between text and parts of other objects (e.g., building edges) - Choice of OBIA algorithms and definition of parameters and thresholds for the generation of a rule set - Choice of appropriate current dataset for object linking - Error estimation and quality assessment of results by replacing control points and regarding the spatial context of objects RQ2.3 Can an approach be developed that is transferable to various large-scale historical maps? Methodology: In this dissertation, transferability is defined as the possibility of applying an existing or developed workflow on a range of other maps without the need for major technical adjustments. The transferability of a workflow is essential, particularly with regard to its further usability. As cartographic styles among historical maps are highly heterogeneous, a fully automated, transferable approach would be fruitless. Therefore, applicable semi-automated processes were developed for the purpose of label and object extraction from large-scale historical maps within Papers II and III. In Paper III, the applicability of the object extraction workflow is demonstrated with the help of alternative large-scale historical maps (see example in Figure 8). Also, the label extraction workflow presented in Paper II was applied to multiple maps (see example in Figure 9). The results were quantified by means of common statistical measures. Results: Rule sets, once constructed for the automatic extraction of objects (cf. Paper III) from a specific input in eCognition, can easily be applied to other maps. Maps having a similarly or less complex visual appearance compared to the initial map only require minor changes in the workflow to achieve a satisfying result. Otherwise, as can be seen from Figure 8, alternative parameters and functions, particularly within the OBIA phase, are to be defined. It must be noted that all subsequent processes depend on the quality of OBIA results. Other steps included in the holistic extraction of objects, such as the initial elimination of labels or the final raster-to-vector conversion, are easily transferable to other maps without or with only minor adjustments of single thresholds. 4.2 Answers to research questions 27 Figure 8: a) The original input map described in chapter 4.1 as well as b) a more complex, largescale historical map (Harvard Map Collection et al., n.d.-b) showing the same spatial extent of the city of Hamburg. The OBIA workflow originally developed for map a) (see result in c)) was applied to b) and delivered less satisfactory results, as can be seen in d) The preceding label extraction process presented in Paper II is deemed transferable as long as labels are detected and recognized by the used tools Strabo and Tesseract. The results show that isolated, straight, and homogeneous labels provide ideal preconditions. After applying Strabo to a number of large-scale historical maps, an average f-score of 58 % was achieved. The f-score considers not only true but also false positives, which can be seen in the lower right of Figure 9 b). Figure 9 juxtaposes the detected labels from two exemplary large-scale historical maps. Figure 9: Detected text image areas from two large-scale historical maps (Harvard Map Collection et al., n.d.-a; Europeana, n.d.) after applying Strabo. Besides true positives, false positives – areas erroneously detected as text – were identified in the map on the right, which has a negative impact on precision and f-score 5 Discussion and outlook 34 An initial evaluation of users’ needs regarding the interpretation and comparison of historical maps helped to specify the research objectives of this dissertation. The results from Paper II are consistent with previous literature referring to rather low text detection and recognition rates. However, the process of label extraction could be improved within the scope of this dissertation by extracting also rotated text, which is still an issue in current research. This research stands out against others by going one step further from text recognition: the spatial assignment of extracted labels within the historical map as well as their verification by means of current toponyms. Whereas past research rather treats labels as disturbing elements when extracting other map objects, an initial label elimination further helped to significantly facilitate the object extraction process in Paper III. A final demonstration of the workflow’s applicability proved beneficial, although a testing with alternative maps showing other cities and semiologies is advisable. Agreeing with many other authors, there is no universal method that can be applied to all historical maps. Especially when regarding monochrome maps, overlaps between objects, labels, map grids, etc. are still an issue. Therefore, large-scale historical maps cannot solely be segmented on the basis of their spectral information for the purpose of semi-automated information extraction. Strengths and limitations In the study of historical maps, potential sources of uncertainties should be considered. Unlike today, maps were not produced based on modern surveying or precise satellite imagery in the past but by means of rather inaccurate surveying or triangulation. Inaccuracies might have also been evolved due to lacking understandings, spatial references, or written records of locations and measures. Therefore, a one-to-one comparison between highly precise maps from present times and approximated historical maps is not straightforward. The user survey conducted in the course of Paper I represents a first attempt to assess and respond to actual challenges and requirements in the investigation of historical maps. However, further research examining the usefulness and applicability of the workflows developed in the context of this thesis should be done. One limitation of the label extraction process presented in Paper II is, for instance, that toponyms of e.g., streets and places should not change significantly over time, which is presupposed for a fuzzy matching between historical and current data. Further restrictions originating from the used tool Strabo concern handwritten and curved labels as well as overlapping map features. Nevertheless, by combining such available tools with minor enhancements, a comprehensive workflow to extract semantic information from large-scale historical maps could be developed. Thereby, the number of true positives was increased, but, at the same time, also false positives emerged. When executing the suggested solution for label extraction with other historical maps, manual user interaction is hardly needed. Certain limitations also arise from Paper III. For example, the usage of proprietary OBIA software impedes a general access to users. The subjective but universal definition of functions and parameters requires trial and error as well as human expert knowledge, which cannot be expected from any type of users. Therefore, the exchange and cooperation between developers and users must be strengthened. When applying object-based image analysis in the context of object extraction, it must be noted that all subsequent steps depend on the quality of the initial results. One of these steps, the matching of different vector datasets based on shape similarities, additionally demands a similar degree of detail, scale, and generalization among these files. Furthermore, there must be at least three geometries matching between two datasets to define control points for an affine transformation. The quality of the resulting spatial transformation further depends on the spatial distribution of these control points throughout the map. 5.2 Outlook 35 Although the general applicability of the presented object extraction workflow is limited, eCognition allows a simple transferability of developed process chains. Unlike pixel-based approaches, which only regard color differences between single pixels, several visual variables may be considered within objectbased image analysis. Thus, objects from typically monochrome or low-contrast historical maps can be extracted. This is additionally facilitated by the preceding elimination of labels. To finally enable a direct comparison of data extracted from a large-scale historical map with a current counterpart, a rough spatial transformation was suggested in Paper III. The demonstrated approach largely prevents a spatial distortion of original shapes within a historical map. 5.2 Outlook This dissertation has demonstrated how semi-automated processes for information extraction from largescale historical maps have the potential to contribute to the investigation of long-term developments in the urban space. The presented solutions show how objects and labels can be extracted in an exemplary way and further used for analyzing, processing, and comparing purposes. However, further investigations are needed in this context. A combination of results from the publications being part of this thesis would be useful to assign semantic descriptions to related map objects (see Figure 11). Thus, datasets from other sources can easily be linked. Figure 11: Vectorized map objects extracted from a large-scale historical map in the course of Paper III and spatially rectified current names of roads, buildings, and water areas, as extracted within the framework of Paper II. Base map is the historical map shown in Figure 5 An additional extraction of roads and places would complete the object extraction from large-scale historical maps. Extracted road names may then be assigned directly to their corresponding geometries. However, Figure 11 shows how roads are often not represented by discrete features but rather in a similar way to figure-ground diagrams (cf. RQ1.3) so that an automated extraction is hardly feasible. Another open question in this context remains how starting and end points of single streets could be defined. Instead, current, point-like road names were spatially rectified and assigned in this thesis. Further research remains to be done concerning the allocation of point-like labels to polygonal roads and the derivation and assignment of original historical names. This also applies to other map objects. To generate 5 Discussion and outlook 36 appropriate databases containing semantic information from both current and historical maps, historical names must be recorded as well. This allows a direct comparison between former and modern designations of e.g., individual roads or buildings. The provision of databases further enables a storage of additional data values such as class assignments, geometrical measures, or information from secondary sources. Improving the comparison of large-scale historical maps, which was a major objective of this thesis, is no straightforward matter because detailed reproductions of reality and precise accuracies were of minor importance in the past (Crom & Heinz, 2016). As investigated in Paper I, it could be helpful to adapt the visual appearance of maps to one another. Comparable to Ory et al. (2017), who generated several in-between representations by interpolation, the semiology, e.g., colors, textures, and line widths, of largescale historical might be applied to current maps or vice versa. According to users of historical maps (cf. Paper I), the manipulation of visual variables would be a conceivable approach to standardize map styles among maps from different ages. The purely visual comparison between historical and current maps may thus be simplified (Kang et al., 2019). In doing so, road features might be of particular interest. While streetscapes usually are linear in current geodata, they rather form polygonal features in historical maps. In terms of future research, an interpretation and evaluation of the results from this dissertation should be conducted. On the one hand, the examination of the performance, robustness, and accuracy of developed workflows is a remaining issue due to the developer’s subjective influence. On the other hand, there is a need for a qualitative results assessment (Lladós et al., 2001). To address the latter, a ground truth can be useful but provides an evaluation basis which solely depends on human perception. Validating results by means of a single and highly subjective benchmark is not recommended. Inaccuracies are common due to deviations originating from e.g., antialiasing or shadows drawn along object boundaries (Heitzler & Hurni, 2020; Lefèvre et al., 2019). Further investigation is therefore needed regarding the geometric accuracies of “positions, distances, areas, and angles of features on the map“ (Jenny & Hurni, 2011, p. 403) in comparison to reality or other, more reliable maps. Additionally, the correct assignment of objects to object classes should be researched in greater depth. In Wu et al. (2022), for instance, uncertainty maps are generated to evaluate the accuracy of segmentation results from historical maps. In addition to those already mentioned, several aspects should be subject to further investigations. First, the quantity of identifiable and extractable labels and objects should be increased in text detection and recognition tools. Second, more research is needed in the investigation of alternative map examples to analyze the applicability of the suggested workflows. An advancement of automatization is a reasonable step in future work with the aim to further reduce manual interaction. Third, the implementation of all processes involved in the presented workflows within an integrated program code is required to establish a comprehensive and stand-alone tool for the (semi-)automated information extraction from large-scale historical maps. In this context, further research should focus on the potential replacement of proprietary software for the entire workflow. In summary, this dissertation can be seen as an important step towards an increased automation regarding the extraction of information from large-scale historical maps. Contributing to the investigation of our past, this research provides a number of novel findings concerning the improved searchability, analyzability, and comparability of historical maps and may stimulate further investigation in this important research area. 37 Literature Ablameyko, S., Bereishik, V., Homenko, M., Lagunovsky, D., Paramonova, N., & Patsko, O. (2002). A complete system for interpretation of color maps. International Journal of Image and Graphics, 2(3), 453–479. https://doi.org/10.1142/s0219467802000767 Ablameyko, S., Bereishik, V., Homenko, M., Paramonova, N., & Patsko, O. (2001). Interpretation of colour maps. A combination of automatic and interactive techniques. Computing & Control Engineering, 12(4), 188–196. https://doi.org/10.1142/S0219467802000767 Andrews, J. H. (2009). Maps in those days. Cartographic methods before 1850. Four Courts Press. Babu, R. M., Srimaiyee, P., & Srikrishna, A. (2010). Text extraction from heterogenous images using mathematical morphology. Journal of Theoretical and Applied Information Technology, 16(1), 39–47. Balletti, C., & Guerra, F. (2009). Image matching for historical maps comparison. e-Perimetron, 4(3), 180–186. Baltsavias, E. P. (2004). Object extraction and revision by image analysis using existing geodata and knowledge: Current status and steps towards operational systems. ISPRS Journal of Photogrammetry and Remote Sensing, 58(3–4), 129–151. https://doi.org/10.1016/j.isprsjprs.2003.09.002 Beconytè, G. (2011, July 3-8). Cartographic styles: Criteria and parameters [Paper presentation]. In A. Ruas. Proceedings of the 25th International Cartographic Conference. 25th International Cartographic Conference, Paris. Benavides, J., & Koster, E. (2006). Identifying surviving landmarks on historical maps. e-Perimetron, 1(3), 194–208. Bertin, J. (1967). Sémiologie graphique. Gauthier-Villars. Bhowmik, S., Sarkar, R., Nasipuri, M., & Doermann, D. (2018). Text and non-text separation in offline document images: a survey. International Journal on Document Analysis and Recognition, 21, 1–20. https://doi.org/10.1007/s10032-018-0296-z Blaschke, T., Hay, G. J., Kelly, M., Lang, S., Hofmann, P., Addink, E., Queiroz Feitosa, R., van der Meer, F., van der Werff, H., van Coillie, F., & Tiede, D. (2014). Geographic object-based image analysis - Towards a new paradigm. ISPRS Journal of Photogrammetry and Remote Sensing, 87, 180–191. https://doi.org/10.1016/j.isprsjprs.2013.09.014 Boutoura, C., & Livieratos, E. (2006). Some fundamentals for the study of the geometry of early maps by comparative methods. e-Perimetron, 1(1), 60–70. Bucha, V., Ablameyko, S., & Pridmore, T. (2005, April 4-6). Semi-automatic extraction and vectorisation of multicoloured cartographic objects [Paper presentation]. IEE International Conference on Visual Information Engineering, Glasgow (pp. 115–120). https://doi.org/10.1049/cp:20050079 Budig, B. (2016, June 8). Towards a pipeline for metadata extraction from historical maps [Paper presentation]. In T. C. van Dijk, & C. Schommer. Exploring Old Maps. International Workshop on Exploring Old Maps, Luxembourg. Literature 38 Budig, B., & van Dijk, T. C. (2015, October 4-6). Active learning for classifying template matches in historical maps [Paper presentation]. In N. Japkowicz, & S. Matwin. Discovery Science. 18th International Conference on Discovery Science, Banff (pp. 33–47). Springer. https://doi.org/ 10.1007/978-3-319-24282-8_5 Cartography Associates. (n.d.-a). David Rumsey. https://www.davidrumsey.com/about/david-rumsey Cartography Associates. (n.d.-b). Collection history. https://www.davidrumsey.com/about/about Chaves, M. (2021, January 21). GLCMs – a great tool for your ML arsenal. Towards Data Science. https://towardsdatascience.com/glcms-a-great-tool-for-your-ml-arsenal-7a59f1e45b65 Chen, Y. L., Hong, Z. W., & Chuang, C. H. (2012). A knowledge-based system for extracting text-lines from mixed and overlapping text/graphics compound document images. Expert Systems with Applications, 39(1), 494–507. https://doi.org/10.1016/j.eswa.2011.07.040 Chiang, Y.-Y. (2010). Harvesting geographic features from heterogeneous raster maps (Order No. 3434411) [Doctoral dissertation, University of Southern California]. ProQuest Dissertations & Theses Global. https://www.proquest.com/dissertations-theses/harvesting-geographic-features-heterogeneous/doc view/847028610/se-2 Chiang, Y.-Y. (2015, November 3-6). Querying historical maps as a unified, structured, and linked spatiotemporal source (Vision paper) [Paper presentation]. In Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems. SIGSPATIAL '15, Seattle (pp. 16:1–16:4). Association for Computing Machinery. https://doi.org/10.1145/2820783. 2820887 Chiang, Y.-Y. (2017, December 16-17). Unlocking textual content from historical maps - Potentials and applications, trends, and outlooks [Paper presentation]. In K. C. Santosh, M. Hagarge, V. Bevilacqua, & A. Negi. Recent Trends in Image Processing and Pattern Recognition. Recent Trends in Image Processing and Pattern Recognition, Bidar (pp. 111–124). Springer. https://doi.org/10.1007/ 978-981-10-4859-3_11 Chiang, Y.-Y., Duan, W., Leyk, S., Uhl, J. H., & Knoblock, C. A. (2020). Using historical maps in scientific studies. Applications, challenges, and best practices. Springer. Chiang, Y.-Y., & Knoblock, C. A. (2014). Recognizing text in raster maps. GeoInformatica, 19(1), 1–27. https://doi.org/10.1007/s10707-014-0203-9 Chiang, Y.-Y., Leyk, S., Honarvar Nazari, N., Moghaddam, S., & Tan, T. X. (2016). Assessing the impact of graphical quality on automatic text recognition in digital maps. Computers and Geosciences, 93, 21–35. https://doi.org/10.1016/j.cageo.2016.04.013 Chiang, Y.-Y., Leyk, S., & Knoblock, C. A. (2014). A survey of digital map processing techniques. ACM Computing Surveys, 47(1), 1:1–1:44. https://doi.org/10.1145/2557423 Christophe, S. (2012, September 16-18). Cartographic styles between traditional and original (towards a cartographic style model) [Paper presentation]. AutoCarto 2012, Columbus. Literature 39 Christophe, S., Duménieu, B., Turbet, J., Hoarau, C., Mellado, N., Ory, J., Loi, H., Masse, A., Arbelot, B., Vergne, R., Brédif, M., Hurtut, T., Thollot, J., & Vanderhaeghe, D. (2016, May 7-9). Map style formalization: Rendering techniques extension for cartography [Paper presentation]. In P. Bénard, & H. Winnemoeller. Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering. Expressive 2016, Lisbon (pp. 59–68). Eurographics Association. Chrysovalantis, D. G., & Nikolaos, T. (2020, March 13). Building footprint extraction from historic maps utilizing automatic vectorisation methods in open source GIS software [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 9–17). Coates, A., Carpenter, B., Case, C., Satheesh, S., Suresh, B., Wang, T., Wu, D. J., & Ng, A. Y. (2011, September 18-21). Text detection and character recognition in scene images with unsupervised feature learning [Paper presentation]. In Proceedings of the 11th International Conference on Document Analysis and Recognition. 11th International Conference on Document Analysis and Recognition, Beijing (pp. 440–445). IEEE Computer Society. https://doi.org/10.1109/ICDAR.2011.95 Crom, W. (2013). Von Impulsen, Kontinuitäten und Brüchen in der Kartengeschichte. In K. Geus & M. Rathmann (Eds.), Vermessung der Oikumene (pp. 321–350). De Gruyter. Crom, W., & Heinz, M. (2016). Karten, Kartographiegeschichte, Geschichtskarten. In L. Busse, W. Enderle, R. Hohls, G. Horstkemper, T. Meyer, J. Prellwitz & A. Schuhmann (Eds.), Clio Guide. Ein Handbuch zu digitalen Ressourcen für die Geschichtswissenschaften (2nd ed.). HumboldtUniversität zu Berlin. https://guides.clio-online.de/guides/sammlungen/karten-kartographie geschichte-geschichtskarten/2016 Dent, B. D., Torguson, J. S., & Hodler, T. W. (2009). Cartography. Thematic map design (6th ed.). McGraw-Hill. Dhar, D. B., & Chanda, B. (2006). Extraction and recognition of geographical features from paper maps. International Journal on Document Analysis and Recognition, 8(4), 232–245. https://doi.org/10.1007/ s10032-005-0010-9 Dodge, M., Kitchin, R., & Perkins, C. (Eds.). (2011). The map reader. Theories of mapping practice and cartographic representation (1st ed.). John Wiley & Sons Ltd. Drapeau, J., Géraud, T., Coustaty, M., Chazalon, J., Burie, J. C., Eglin, V., & Bres, S. (2017, November 9-10). Extraction of ancient map contents using trees of connected components [Paper presentation]. In A. Fornés, & B. Lamiroy. Graphics Recognition. Current Trends and Evolutions. 12th IAPR International Workshop, GREC 2017, Kyoto (pp. 115–130). Springer. https://doi.org/ 10.1007/978-3-030-02284-6_9 Ehlers, M:, & Schiewe, J. (2012). Geoinformatik. WBG. Ekim, B., Sertel, E., & Erdem Kabadayı, M. (2021). Automatic road extraction from historical maps using deep learning techniques: A regional case study of Turkey in a German world war II map. ISPRS International Journal of Geo-Information, 10(8), 492. https://doi.org/10.3390/ijgi10080492 El-Hussainy, M. S., Baraka, M. A., & El-Hallaq, M. A. (2011). A methodology for image matching of historical maps. e-Perimetron, 6(2), 77–95. Literature 40 Europeana. (n.d.). Plan von Hamburg, 1:11 000, Lithographie, 1834. Europeana. Retrieved February 12, 2019, from https://www.europeana.eu/portal/de/record/2048415/item_WAZONILNG5GABKG RM2CH2OYB5TJN25GG.html Fleet, C., Kowal, K. C., & Přidal, P. (2012). Georeferencer: Crowdsourced georeferencing for map library collections. D-Lib Magazine, 18(11/12). https://doi.org/10.1045/november2012-fleet Frank, R., & Ester, M. (2006, July 12-14). A quantitative similarity measure for maps [Paper presentation]. In A. Riedl, W. Kainz, & G. A. Elmes. Progress in Spatial Data Handling. 12th International Symposium on Spatial Data Handling, Vienna (pp. 435–450). Springer. https://doi.org/10.1007/3-540-35589-8_28 Garcia-Molsosa, A., Orengo, H. A., Lawrence, D., Philip, G., Hopper, K., & Petrie, C. A. (2021). Potential of deep learning segmentation for the extraction of archaeological features from historical map series. Archaeological Prospection, 28(2), 187–199. https://doi.org/10.1002/arp.1807 Gede, M., Árvai, V., Vassányi, G., Supka, Z., Szabó, E., Bordács, A., Varga, C. G., & Irás, K. (2020, March 13). Automatic vectorisation of old maps using QGIS – tools, possibilities and challenges [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 37–43). Gladstone, C. S., Gardiner, A., & Holland, D. (2012, May 7-9). A semi-automatic method for detecting changes to Ordnance Survey topographic data in rural environments [Paper presentation]. In Proceedings of the 4th GEOBIA. 4th International Conference on Geographic Object-Based Image Analysis, Rio de Janeiro (pp. 396–401). Gobbi, S., Ciolli, M., la Porta, N., Rocchini, D., Tattoni, C., & Zatelli, P. (2019). New tools for the classification and filtering of historical maps. International Journal of Geo-Information, 8(10), 455. https://doi.org/10.3390/ijgi8100455 Goldstein, E. B. (2008). Cognitive psychology: Connecting mind, research, and everyday experience (2nd ed.). Thomson Wadsworth. Goss, J. (1994). KartenKunst. Die Geschichte der Kartographie. Westermann. Groom, G., Levon, G., Svenningsen, S., & Perner, M. L. (2020, March 13). Historical maps – Machine learning helps us over the map vectorisation crux [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 89–98). Hake, G., Grünreich, D., & Meng, L. (2002). Kartographie. Visualisierung raum-zeitlicher Informationen (8th ed.). De Gruyter. Harley, J. B. (1987). The map and the development of the history of cartography. In J. B. Harley & D. Woodward (Eds.), The history of cartography. Cartography in prehistoric, ancient, and medieval Europe and the Mediterranean (Vol. 1, pp. 1–42). The University of Chicago Press. Harvard Map Collection, Harvard College Library, Lindley W., engineer., Society for the Diffusion of Useful Knowledge (Great Britain), Cox G., publisher., & Davies B. R. (n.d.-a). Hamburg, Germany, 1853. Princeton University Library. Retrieved April 26, 2019, from https://maps.princeton.edu/ catalog/harvard-g6299-h3-1853-l5 Literature 41 Harvard Map Collection, Harvard College Library, Mirbeck, C. L. B., & Baker, B. (n.d.-b). Hamburg, Germany, 1803. Princeton University Library. Retrieved April 26, 2019, from https://maps. princeton.edu/catalog/harvard-g6299-h3-1803-m5 Hay, G. J., & Castilla, G. (2008). Geographic object-based image analysis (GEOBIA): A new name for a new discipline. In T. Blaschke, S. Lang, & G. J. Hay (Eds.), Object-based image analysis. Lecture notes in geoinformation and cartography (pp. 75–89). Springer. Heitzler, M., & Hurni, L. (2020). Cartographic reconstruction of building footprints from historical maps: A study on the Swiss Siegfried map. Transactions in GIS, 24(2), 442–461. https://doi.org/10.1111/tgis.12610 Herold, H. (2018). Geoinformation from the past: Computational retrieval and retrospective monitoring of historical land use. Springer. Herrault, P.-A., Sheeren, D., Fauvel, M., & Paegelow, M. (2013, May 14-17). Automatic extraction of forests from historical maps based on unsupervised classification in the CIELab color space [Paper presentation]. In D. Vandenbroucke, B. Bucher, & J. Crompvoets. Geographic Information Science at the Heart of Europe. The 16th AGILE International Conference on Geographic Information Science, Leuven (pp. 95–112). Springer. Höhn, W., Schmidt, H.-G., & Schöneberg, H. (2013, July 22-26). Semiautomatic recognition and georeferencing of places in early maps [Paper presentation]. In J. S. Downie, R. H. McDonald, T. W. Cole, R. Sanderson, & F. Shipman. Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries. 13th ACM/IEEE-CS Joint Conference on Digital Libraries, Indianapolis (pp. 335–338). Association for Computing Machinery. https://doi.org/10.1145/2467696.2467734 Höhn, W., & Schommer, C. (2017, August 8-11). Georeferencing of place markers in digitized early maps by using similar maps as data source [Paper presentation]. In R. Lewis, C. Raynor, D. Forest, M. Sinatra, & S. Sinclair. Digital humanities 2017: Conference abstracts. Digital Humanities 2017, Montreal (pp. 473–475). Howe, N. R., Weinman, J., Gouwar, J., & Shamji, A. (2019, November 5). Deformable part models for automatically georeferencing historical map images [Paper presentation]. In F. Banaei-Kashani, G. Trajcevski, R. H. Güting, L. Kulik, & S. Newsam. SIGSPATIAL ’19: Proceedings of the 27th ACM SIGSPATIAL international conference on advances in geographic information systems. 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Chicago (pp. 540–543). https://doi.org/10.1145/3347146.3359367 Hussain, M., Chen, D., Cheng, A., Wei, H., & Stanley, D. (2013). Change detection from remotely sensed images: From pixel-based to object-based approaches. ISPRS Journal of Photogrammetry and Remote Sensing, 80, 91–106. https://doi.org/10.1016/j.isprsjprs.2013.03.006 Imhof, E. (1972). Band 10: Thematische Kartographie. In E. Obst (Ed.). Lehrbuch der allgemeinen Geographie. De Gruyter. International Organization for Standardization. (2022). Intelligent transport systems (ITS) – Location referencing for geographic databases – Part 1: General requirements and conceptual model (ISO Standard No. ISO 17572-1:2022(en)). https://www.iso.org/obp/ui/#iso:std:iso:17572:-1:ed-3: v1:en Literature 42 Iosifescu, I., Tsorlini, A., & Hurni, L. (2016). Towards a comprehensive methodology for automatic vectorization of raster historical maps. e-Perimetron, 11(2), 57–76. Iwanowski, M., & Kozak, J. (2012). Automatic detection of forest regions on scanned old maps. Przegląd Elektrotechniczny, 88(4a), 249–252. Jenny, B., & Hurni, L. (2011). Studying cartographic heritage: Analysis and visualization of geometric distortions. Computers and Graphics, 35(2), 402–411. https://doi.org/10.1016/j.cag.2011.01.005 Jessop, M. (2006). Promoting cartographic heritage via digital resources on the web. e-Perimetron, 1(3), 246–252. Jiao, C., Heitzler, M., & Hurni, L. (2020, March 13). Extracting wetlands from Swiss historical maps with convolutional neural networks [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 31–36). Kang, Y., Gao, S., & Roth, R. E. (2019). Transferring multiscale map styles using generative adversarial networks. International Journal of Cartography, 5(2–3), 115–141. https://doi.org/10.1080/ 23729333.2019.1615729 Karakuyu, M. (2011). A new approach to analyzing historical urban growth of Ottoman cities. Manisa case study. Historical Methods: A Journal of Quantitative and Interdisciplinary History, 44(3), 131–138. https://doi.org/10.1080/01615440.2011.568927 Kasturi, R., & Alemany, J. (1988). Information extraction from images of paper-based maps. IEEE Transactions on Software Engineering, 14(5), 671–675. https://doi.org/10.1109/32.6145 Kenley, R., Yeo, R., & Harfield, T. (2019). Three location referencing method families: a new theoretical framework [Paper presentation]. In IOP Conference Series: Materials Science and Engineering, 512. 10th Malaysian Road Conference & Exhibition 2018, Selangor. http://dx.doi.org/10.1088/1757899X/512/1/012059 Kim, J. O., Yu, K., Heo, J., & Lee, W. H. (2010). A new method for matching objects in two different geospatial datasets based on the geographic context. Computers and Geosciences, 36(9), 1115–1122. https://doi.org/10.1016/j.cageo.2010.04.003 Kim, N. W., Lee, J., Lee, H., & Seo, J. (2014). Accurate segmentation of land regions in historical cadastral maps. Journal of Visual Communication and Image Representation, 25(5), 1262–1274. https://doi.org/10.1016/j.jvcir.2014.01.001 Knowles, A. K. (Ed.). (2008). Placing history: How maps, spatial data, and GIS are changing historical scholarship. ESRI Press. Larcher, V., & Piovan, S. (2018). The use of colours in historical atlases. Cartographica: The International Journal for Geographic Information and Geovisualization, 53(2), 115–128. https://doi.org/10.3138/ cart.53.2.2017-0012 Laumer, D., Heitzler, M., & Hurni, L. (2020, March 13). A semi-automatic label digitization workflow for the Siegfried map [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 55–62). Literature 43 Laycock, S. D., Brown, P. G., Laycock, R. G., & Day, A. M. (2011). Aligning archive maps and extracting footprints for analysis of historic urban environments. Computers and Graphics, 35(2), 242–249. https://doi.org/10.1016/j.cag.2011.01.002 Lefèvre, S., Sheeren, D., & Tasar, O. (2019). A generic framework for combining multiple segmentations in Geographic Object-based image analysis. ISPRS International Journal of Geo-Information, 8(2), 70. https://doi.org/10.3390/ijgi8020070 le Riche, M. (2020, March 13). Identifying Building Footprints in Historic Map Data using OpenCV and PostGIS [Paper presentation]. In K. Irás. Automatic Vectorisation of Historical Maps. International workshop organized by the ICA Commission on Cartographic Heritage into the Digital, Budapest (pp. 18–30). Li, L., Nagy, G., Samal, A., Seth, S., & Xu, Y. (2000). Integrated text and line-art extraction from a topographic map. International Journal on Document Analysis and Recognition, 2(4), 177–185. https://doi.org/10.1007/PL00021524 Li, Z., Chiang, Y.-Y., Banisetti, S., & Kejriwal, L. (2018). strabo-text-recognition-deep-learning 0.67 [GitHub repository]. https://github.com/spatial-computing/strabo-text-recognition-deep-learning Lin, H., & Chiang, Y.-Y. (2017). SRC: Automatic extraction of phrase-level map labels from historical maps. SIGSPATIAL Special, 9(3), 14–15. https://doi.org/10.1145/3178392.3178400 Liu, T., Xu, P., & Zhang, S. (2019). A review of recent advances in scanned topographic map processing. Neurocomputing, 328, 75–87. https://doi.org/10.1016/j.neucom.2018.02.102 Livieratos, E. (2006). On the study of the geometric properties of historical cartographic representations. Cartographica, 41(2), 165–176. https://doi.org/10.3138/RM86-3872-8942-61P4 Lladós, J., Valveny, E., Sánchez, G., & Martí, E. (2001, September 7-8). Symbol recognition: Current advances and perspectives [Paper presentation]. In D. Blostein, & Y.-B. Kwon. Graphics Recognition: Algorithms and Applications. Lecture Notes in Computer Science (Vol. 2390). IAPR International Workshop on Graphics Recognition, Kingston (pp. 104–127). Springer. https://doi.org/10.1007/3540-45868-9_9 Longley, P. A., Goodchild, M. F., Maguire, D. J., & Rhind, D. W. (2015). Geographic information science and systems (4th ed.). Wiley. Loran, C., Haegi, S., & Ginzler, C. (2018). Comparing historical and contemporary maps - a methodological framework for a cartographic map comparison applied to Swiss maps. International Journal of Geographical Information Science, 32(11), 2123–2139. https://doi.org/10.1080/ 13658816.2018.1482553 Losang, E. (2015, August 23-25). GeoImaginaries – Historic maps and their perception - a critical analysis [Paper presentation]. In C. R. Sluter, C. B. M. Cruz, S. P. Camboim, L. S. Delazari, M. D. C. Fernandes, R. S. D. Barros, H. Firkowski, & P. Márcio. Proceedings of the 27th International Cartographic Conference. 27th International Cartographic Conference, Rio de Janeiro. Medyńska-Gulij, B., & Żuchowski, T. J. (2018). European topography in eighteenth-century manuscript maps. Bogucki Wydawnictwo Naukowe. Vol.:(0123456789) 1 3 KN - Journal of Cartography and Geographic Information https://doi.org/10.1007/s42489-019-00016-0 Empirical Study foraDeployment ofaMethodology forImproving theComparability Between Historical andCurrent Maps IngaSchlegel1 Received: 13 March 2019 / Accepted: 25 April 2019 © The Author(s) 2019 Abstract In the investigation of urban development over centuries, the comparison of appropriate maps forms an essential component. The aim of this project is an improvement of an effective and intuitive comparison of historical and current map features. An adjustment of uniform visual variables to individual maps is therefore suggested. An appropriate framework presenting potential solutions for the deployment of a new methodology is based on the analyzed users’ demands. These requirements were identified and evaluated with the aid of purposive sampled experts interviewed with a pencil and paper questionnaire. Two major challenges concerning the comparison between historical and current maps were revealed in a statistical evaluation: a general lack of technical tools and great varieties in semiology. The familiarity with their semiology has the greatest effect on the identification and distinction of map features. Therefore, an adaption of color composition, textures, and labels seems crucial in particular. Various approaches such as feature extraction or similarity measures to meet the mentioned challenges are suggested for future research. Keywords Historical map· Empirical study· Semiology· Visual variables Empirische Studie zur Entwicklung einer Methodik für eine bessere Vergleichbarkeit zwischen historischen und aktuellen Karten Abstrakt In der Erforschung von urbanen Entwicklungen über Jahrhunderte ist der Vergleich entsprechender Karten essentieller Bestandteil. Das Ziel dieses Projektes ist eine Optimierung für einen effektiven und intuitiven Vergleich zwischen historischen und aktuellen Geoobjekten. Zu diesem Zweck wird eine Vereinheitlichung der visuellen Variablen (Semiologie) von individuellen Karten angestrebt. Ein Überblick, der mögliche Lösungsansätze zur Entwicklung einer neuen Methode hervorbringt, basiert auf erhobenen Nutzeranforderungen. Diese wurden im Rahmen eines Papierfragebogens, welcher von einer Zielstichprobe aus Experten beantwortet wurde, ermittelt. Eine statistische Auswertung brachte zwei große Herausforderungen beim Vergleich von historischen mit aktuellen Karten zum Vorschein: ein allgemeiner Mangel an technischen Werkzeugen sowie große Variationen in der Semiologie. Auf das Identifizieren und Differenzieren von Geoobjekten hat die Vertrautheit mit bekannter Semiologie den größten Einfluss. Insbesondere eine Anpassung der Farbzusammensetzung, Texturen und Beschriftungen ist daher wesentlich. Verschiedene Lösungsansätze wie Merkmalsextraktion oder Ähnlichkeitsmaße werden für künftige Forschungsvorhaben empfohlen, um die genannten Herausforderungen zu bewältigen. Schlüsselwörter Historische Karte· Empirische Studie· Semiologie· Visuelle Variablen 1 Introduction In the recent past, current geodata styled in a historical appearance is often seen on social media or in new-fashioned atlases. Modern city maps such as of London (see Fig.1), * Inga Schlegel [email protected] 1 Lab forGeoinformatics andGeovisualization (g2lab), HafenCity University Hamburg, Hamburg, Germany KN - Journal of Cartography and Geographic Information 1 3 Paris, or Hamburg are visually transformed so that at first glance, they appear to be from and represent past centuries. Assuming an appealing appearance to be the motivation for designers or printmakers to produce current maps in a historic style, the compliance of stylistic and especially cartographic conventions is of secondary importance as the use of colors, contours, textures, and illustrated details relies on highly subjective decisions. But what if such maps are applied in further scientific work—to measure distances, ascertain house owners, former street names, or the progress of urban structures? As processing and analysis of historical maps is not as easy compared to their more recent counterparts, archivists, librarians, historians, urban planners, cartographers, and geographers are frequently confronted with challenges while comparing current with historical geodata portraying one and the same section of a city. In such a comparison process, long-term spatiotemporal urban transformations such as the development of demography, migration flows, or trading and road networks influencing urbanization processes are supposed to become visible. A possibility for extracting designations, addresses, or further database-supported information from simple (paper) images representing historical maps is therefore required. Owing to the lack of alternatives, users yet partly rely on very unconventional methods (own evaluation, see Sect.4.3). As a first step of an overall project, the present study aims at setting foundations for the deployment of a new methodology to compare historical and current maps in a more effective way. This is to be achieved by applying uniform rules of representation to gain new knowledge in the subject of urban development. Only a handful of scientific research efforts take up a uniform adjustment of visual characteristics to better compare current with historical geodata. The study presented here regards the users’ demands and the potential for an improvement concerning an intuitive comparability on the basis of the maps’ semiological characteristics. This paper consequently first, clarifies the terms of and connection between historical maps and semiology, second, reports on and evaluates the outcomes of an on-topic user study implemented in terms of a requirement analysis, and third, based on its results, points out the rationale of the proposed following investigations. 2 Review ofRelated Literature andApproaches From the esthetic point of view, numerous examples of current geodata having a historical appearance can be found. Map designer and illustrator Mike Hall creates maps in a historical manner, particularly inspired by sixteenth up to eighteenth century mapmakers such as Willem Blaeu, John Ogilby, or John Rocque (M. Hall, personal communication, October 22, 2018). Besides portraying the physical world, Fig. 1 Map of modern London inspired by John Rocque’s 1746 Map of London, drawn up by map designer Mike Hall (Hall 2016a) KN - Journal of Cartography and Geographic Information 1 3 he also designs city maps such as his Map of modern London inspired by John Rocque (Hall 2016a) (see Fig.1) or Maps of Glasgow for Denise Mina’s “The Long Drop” (Hall 2016b). Nelson (2016) reports on the work of cartographer Christopher Wesson who transferred visual characteristics (colors, patterns, symbols, labels, and generalization) of an official ordnance survey map from 1801 to recent geodata of London. This mid-scale map was produced at a high level of detail mainly with the help of geographic information systems and image processing software. Similar works by Kay (2016) use OpenStreetMap to map the city of Edinburgh in the style of the early twentieth century. Also, Wellingtons Travel Co. (2017) illustrates a cityscape from 2012 appearing to be historical in the course of their hand drawing The Grand Map of London. While the standard of design in digital (web) maps is often criticized as low, users point out a frequent visual overload of information in historical maps (Beconytè 2011; Christophe etal. 2016; Ory etal. 2013). Analyzing their historical content turns out to be complex and time-consuming. Also, the visual merger of labels with other map elements makes it difficult to differentiate. As a consequence of their heterogeneity due to varying manufacturing eras, different authors and drawing styles, old maps drawn by hand have considerable limitations regarding their machine readability. Besides these consequences of manual production, also technical challenges become apparentsuch as the age of the maps themselves or such arisen from scanning (e.g., blurring or pseudocolor), inducing a low image quality(Leyk and Boesch 2010). Though visually appealing, limitations of historical maps must be mentioned and considered as well. Nevertheless, Field’s (2018) statement coincides with the results of the study presented in Sect.4 that historical maps are generally favored over current ones when only considering their esthetic point of view. This may be explained not only by nostalgic or fashion-oriented reasons, but also by major efforts needed in intellectuality and time for the production of maps in earlier times. People still have a high degree of confidence in ancient maps. Unlike today, maps used to be powerful instruments for the communication of meanings in the past. Nowadays, various maps may be produced and disseminated in a minimum of time—regardless of their validity. For the reverse process—making maps representing spatial information on historical circumstances in a current style—a majority is found solely on a very large scale or seen as artistically playful. However, series of unified representations of the spatial development of antique cities such as Rome, Athens, or Jerusalem over various centuries are often seen in common school atlases (e.g., “Rom—Antike Metropole—Bauwerke” (n.d.), W.W. Norton and Company Inc. (2010)). Aiming at the comparison between contemporary geodata and historical counterparts, various existing tools and approaches are known and applied by different users. One of the most common, but also most time-consuming methods is placing georeferenced historical maps on top of current satellite imagery or vector data—occasionally including functions to define different levels of opacity. The geoportals of Klokan Technologies GmbH (2017), of the Archives nationales de Luxembourg & Musée d’Histoire de la Ville de Luxembourg (n.d.) or of various municipal administration agencies are only a few following this approach. In terms of city maps, a common way to compare corresponding geographic features or locations, but different cartographic styles are so-called side by side viewers. One example is provided by the National Library of Scotland (n.d.b): official topographic maps from England are placed side by side with scanned and georeferenced equivalents from former times and synchronized with another, while users pan one of the maps (O’Brien 2014). Other overlay methodologies are suggested by sliding map comparisons of the University of Minnesota (n.d.) (see Fig.2, left) or virtual and interactively moveable magnifying glasses showing historical map extracts on the base of their current counterparts [e.g., National Library of Scotland (n.d.a) (see Fig.2, middle) and Geiling and Esri (2013)]. A superimposed printing of identical city map sections at the present time on one hand, as well as from 1800 on the other hand, was produced on behalf of the Landesbetrieb Fig. 2 Existing tools to compare historical with current geodata by the University of Minnesota (n.d.) (left), the National Library of Scotland (n.d.a) (middle), and the Landesbetrieb Geoinformation und Vermessung Hamburg (2014) (right) KN - Journal of Cartography and Geographic Information 1 3 Geoinformation und Vermessung Hamburg (2014) (Fig.2, right). Further attempts on automated style conversion between various maps have already been made. Neural networks, as used, for instance, in self-controlled image tagging or userspecific product recommendations, could be used for an image classification with the help of trained images (Deshpande, 2016). This shape recognition may be implemented via decision tree, filter-based, or statistical approaches. Another possibility of unifying two map styles by interpolating their colors and line widths into average values is described by Ory etal. (2017). Besides the mentioned approaches, several others such as the modification of the data’s visual portrayal via Styled Layer Descriptors and Symbology Encoding (Christophe etal. 2015, 2016) or the recognition of geometries and their spatial relationships (Gross 1994, as cited in Liu 2004) still cause problems when faced with hand-drawn lines and textures as well as overlapping features. In terms of an effective extraction of information derived from historical maps—on various objects, real locations, or metadata—research is overdue. None of the aforementioned approaches considers a transfer or derivation of any kind of ancillary information. Especially, suggestions for a universal approach are missing so that an interoperability between different map styles cannot be given yet (Budig and van Dijk 2015; Christophe etal. 2016). Also, Field (2018, p.323) confirms that “[…] replicating a[n] historical map with upto-date information is an entirely valid approach”. However, existing algorithms for extracting semantical information from bitmap images like historical maps are insufficient. An optimal balance between different ways of representation of maps needs to be achieved (Budig and van Dijk 2015; Christophe etal. 2016; Leyk and Boesch 2010). Setting out a framework to meet these challenges represents a further major objective addressed in Sect.5.1 of this paper. 3 Terminology 3.1 Historical Maps As different interpretations exist, a historical map in this paper describes a reduced and simplified representation of early geographic characteristics and structures produced in the past (based on Hake etal. 2002). The boundary between the aforementioned and current maps is frequently drawn around the year 1850 (own evaluation, see Sect.4.3). This estimation may be traced back to major developments in the fields of mathematics and technology in the late 19th century. At that time, cartography profited from innovative methods in terms of accuracy such as the triangulation using theodolites (Thompson 2017). The term ‘historical’ is described with reference to past events or phenomena as well as to reproductions in historical presentations by different dictionaries such as The American Heritage dictionary of the English language (Historical 2018) and Merriam-Webster (Historical n.d.). Hake etal. (2002) characterize maps from former times generally by a great age and obsolescence. These are often replaced by newer, edited versions and adapted to a modern way of presentation. A more blurred boundary between historical and current maps must therefore be assumed. 3.2 Semiology In this study, ‘semiology’ is referred to as the sum of visual variables to be perceived, recognized, and differentiated by an observer of a map (Ory etal. 2017). To visually match geographic features between historical and current maps, semiology is considered crucial in terms of this project. Besides an appropriate structure, scale, and generalization, it is also its graphical representation contributing to an intuitive understanding of a map. The latter mainly consists of graphical elements (points, lines, and polygons) as well as composite signs (signatures, halftones, diagrams, and fonts) representing coded information. Variations in graphical (Hake etal. 2002) or rather visual variables (Slocum etal. 2009) being applied to a map’s graphical representation serve not only an esthetic purpose, but also enable the differentiation of qualitative and quantitative contents (Roth 2017). According to Bertin (1973), visual variables in cartography are limited to size, shape, texture, orientation, location, color value (brightness), and hue. Morrison (1974) additionally suggests color saturation and arrangement. Due to missing variables concerning uncertainties, MacEachren (1995) supplements crispness, resolution, and transparency (Hake etal. 2002; Roth 2017; Slocum etal. 2009). Geographic features are represented by one or more visual variables. For a unique distinction and differentiation of cartographic content, visual variables such as the color black for buildings or rails, green for vegetation, or blue for water bodies are utilized (Hake etal. 2002; Larcher and Piovan 2018; Ory etal. 2015). In the special case of historical maps, water bodies for instance are often depicted with parallel dashed lines decreasing in their proximity and stroke width with an increasing distance as seen from the shore (Huffman 2010). 4 User Study 4.1 Aim With the aim of analyzing present needs and requirements, a user study concerning the comparability between historical KN - Journal of Cartography and Geographic Information 1 3 and current maps was conducted. Considering the following key questions, a statistical evaluation was performed: • What are the major challenges in historical maps and their comparison to current counterparts (see Sect.4.3.1)? • What are the most common topics in such comparison processes (see Sect.4.3.2)? • Which explicit map types are (not) suitable for common tasks the users are confronted with in their everyday work? Which semiological characteristics are stated (not) being suitable regarding these tasks for each map type (see Sect. 4.3.3)? In terms of contents, the focus in this study is on geospatial information most comparable to topographic city maps, both for recent and ancient analogs. Besides an inquiry for general challenges with historical maps and their comparison with current counterparts, explicit topographical maps representing the city of Hamburg were examined concerning their readability as well as the users’ intuition regarding semiological characteristics. 4.2 Setup As the interpretation of and between historical and current maps depends on different experiences from various user groups (Groupe μ 1992, as cited in Ory etal. 2015), 58 German archivists, librarians, historians, urban researchers, publishers, curators and similar experts were asked for survey participation. Due to the specified requirement of a regular study of historical as well as current geodata, they were deliberately selected. The interviewed target group should therefore be able to read, perceive, and interpret this data. A pencil and paper questionnaire including brief instructions was delivered to the respondents by mail. In total, 22 questions were formulated as predominantly closed and semi-open questions. Only few open questions are included in the survey. This choice was made to enable a number of qualitative evaluations in addition to a quantitative analysis of results. 4.3 Results After completion of the survey, a response rate of 57% (n = 31) was achieved. All gained information was anonymized, standardized and stored in a database to facilitate a statistical evaluation. Due to the small sampling and further non-response, a representativity in terms of demography (such as age, gender, and occupation group) was not possible. As the actual participant group corresponds to the desired user group selected by purposive sampling, demographic criteria are not of primary importance. 4.3.1 Challenges withHistorical Maps andTheir Comparison withCurrent Counterparts In juxtaposing historical with current maps, distinct objects represent frequent challenges considering their identifiability and comparability as well as the derivation of further information. All respondents have been faced with these difficulties at least once in their work routine. In this context, an important reason mentioned by some of them are distortions caused by cartographic processes or regarding the presented state of reality. For half of them, these difficulties are due to great semiological varieties. To facilitate the technical effort, 20% of interviewees use web tools such as Klokan’s Georeferencer (Klokan Technologies GmbH 2017) or similar map viewers to georeference, overlay, or view maps side by side. Even more respondents managewith Desktop-GIS (37%) or related tools and methods such as image processing software or they meet their needs for an analog comparison (40%). Among the different user groups, librarians, archivists, and historians rarely make use of existing tools and software for a comparison between historical and current maps. Instead, especially librarians prefer using historical paper maps in combination with printed or digital, georeferenced maps representing a current urban image (see Fig.3). Whereas one quarter of the surveyed librarians compared spatial data with each other, even none of the archivists do so. Digital and georeferenced data on present circumstances are used almost exclusively by map experts such as cartographers. They utilize this for a juxtaposition with analog, digitized, as well as georeferenced historical data. 4.3.2 Content ofMap Comparisons Regarding the content, more than half of the respondents stated that they regularly (at least once a month, see Fig.4) compare certain geoobjects—more precisely buildings and roads—between historical and current equivalents. As these two object types make up a rough urban structure, they seem to be the most relevant features for the majority of respondents. On the contrary, changes of water or vegetation areas over time as well as accompanying texts are of minor interest. In the following course of this project, a focus will therefore be placed on the semiological characteristics of buildings and roads as well as on their context within the urban structure. 4.3.3 Different Semiological Characteristics forDifferent Tasks With the aid of explicit examples—both showing historical and current maps—the interviewees were asked for the most appropriate representations concerning various common KN - Journal of Cartography and Geographic Information 1 3 tasks (see Fig.5). The tasks include the identification and the differentiation of geographic features such as buildings, roads, water bodies, and vegetation, as well as the visual recognition of an urban structure. Among current maps, a generally high suitability of the representation of OpenStreetMap is noticeable (see Fig.5, example 1). According to the study participants, it represents the most appropriate map for all mentioned tasks except the identification and differentiation of buildings. Color composition as well as the line width constitutes the deciding factors for a quick and intuitive identification of streets, water bodies, and vegetation and, in particular, for distinguishing different types of roads. Against expectations, this choice does not seem to depend on users’ habits or the familiarity with map services: unlike the presentation of OpenStreetMap, Google Maps performs poorly for each of the mentioned tasks (Fig.5, example 5). Instead, a local map design emerged as the most efficient concerning the identification of buildings and their distinction between different types (Fig.5, example 2). In this map, differing building types are represented by varying texture patterns; furthermore, public buildings clearly differ from others in color. Labels indicating house numbers or abbreviations, to designate, for instance, public institutions, appear to be a key component for an intuitive recognition of map features. Besides labels, textures as well as the color composition of geometries can therefore be seen as major semiological elements enhancing the recognition of different objects in maps. Also among the historical maps shown in Fig.5 (examples 6–9), the applicability of one representation is salient. Being the only colored historical map, the color composition of example 6 represents the most appropriate semiological component facilitating the perception of different geoobjects. Although it might seem that solely water bodies differ substantially from other features due to their divergent color hue, color composition is also considered to be applicable for the identification and differentiation of buildings and roads. For buildings, the variance of color values seems to be crucial, whereas the decisive factor for roads may be solely determined by the high-contrast overall presentation of the map. As the presented historical maps—apart from example 6—are grayscale images, the perception of their geoobjects based on color composition is correspondingly considered minimal. Fig. 3 Share of resources used for both historical and current maps for their mutual comparison by librarians (left) and cartographers (right) Paper Digitized without georeference Digitized without georeference Digitized without georeference Digitized and georeferenced Digitized and georeferenced Digitized and georeferenced Historical maps Librarians Cartographers Current maps Historical maps Paper Paper Less frequent More than 2 times per week Several times per year Once per week 1 to 3 times per month Never Buildings Roads Waterbodies Vegetation Urban structure Accompanying text Fig. 4 Interviewees’ frequencies (relative) of comparing specific features, structures, or content between historical and current maps KN - Journal of Cartography and Geographic Information 1 3 According to the majority of interviewees, the applicability of map example 7 for the identification of buildings is comparable to the one of example 6. Contributing to a high degree of recognizability of roads in the case of example 7 are rather the labels than the color composition. Furthermore, differences in textures (e.g., between parallel and contour hatching, stippling, and blending) and line widths are mentioned for an intuitive recognition of buildings and streets in map examples 6–8 of Fig.5. However, in examples 7–9, the perception of water bodies as well as vegetation is hardly feasible as related familiar semiology is not assigned. According to human intuition, visual variables such as the color green standing for vegetation and blue for water bodies usually serve to identify the appropriate features. This conclusion is also supported by the lack of color for vegetation areas in map examples 2 and 6 in Fig.5. As can be seen from example 9 of Fig.5, an unintuitive representation preventing recognizing and distinguishing map features may be induced by overlaying grids, missing contrast throughout the entire map, as well as insufficient labels. With the aim of explaining a relationship between one dependent (e.g., identifying buildings) and several independent variables (color composition, line width, texture, symbols, and labels), a logistic regression analysis was performed. As results vary considerably, a general statement cannot be made regarding the impact of visual variables on the probability that one of the mentioned tasks can be performed intuitively. 4.4 Summary By conducting a first needs assessmentamong appropriate user groups, major challenges in working with historical maps could be identified. Also,considerable difficulties regarding the comparison process between historical and current maps were presented in detail. Color composition Line width Texture Symbols Labels Ide of buildings ds ation of vegetation of building types erentiation of road types 52143 7698 Fig. 5 Map examples of current (top) and historical (bottom) maps shown in the user study for an evaluation of their applicability with respect to the identification of buildings, roads, water bodies, and vegetation, as well as to the differentiation of building types and road types in consequence of color composition, line width, texture, symbols, and labels (see diagrams). Top, from left to right: (1) OpenStreetMap contributors; (2) Freie und Hansestadt Hamburg; (3) Esri etal.; (4) ©2018 Microsoft Cooperation ©2018 HERE; (5) ©2018 GeoBasis-DE/BKG (©2009); Bottom, from left to right: (6) Harvard University (n.d.); (7) Terstegge (n.d.); (8) Europeana (n.d.); (9) © SLUB/Kartensammlung (2018) KN - Journal of Cartography and Geographic Information 1 3 Firstly, besides perspective distortions or unrealistic presentations, variations in semiology make it difficult to distinguish between map objects. To collate former and present buildings and roads, the overall appearance of a map seems to have a major impact. While contrasting color values best serve for demarcating vegetation areas and water bodies, additional textures may be helpful to distinguish buildings, roads, and their particular types. Labels designating various map features seem to be advantageous for their intuitive recognition predominantly in colorless maps. The distinction of map objects such as buildings, roads, vegetation areas, and water bodies does not appear to depend on a map’s familiarity, but rather on the awareness of visual variables being assigned to corresponding objects. A second lesson—according to actual users such as archivists, librarians, historians, urban planners, cartographers, and geographers—is a current lack of helpful tools and instruments for an intuitive comparison between historical and current maps. 5 Future Research andConclusion 5.1 Derived Project Concept forFurther Investigations Existing concepts solely considering partial aspects of the defined problem area (see Sect.2) shall be optimized to provide a possible holistic solution. Appropriate approaches to facilitate the comparison process between historical and current maps are suggested in this section. These will be examined and implemented in the further course of the overall project. With the aim of making historical geodata as editable and applicable as its current counterpart, several methods for feature extraction and classification may be applied in a first stage. As a result, geometric shapes (especially lines and polygons) can be derived from a historical map and assigned to different feature classes (e.g., roads and buildings as well as their subgroups), thus improving the information content compared to the original bitmap image. • Similarly to the procedure with satellite imagery, image segmentation may be used to separate a bitmap image into patches having internally consistent properties (e.g., in size, shape, and texture). An object-based image analysis even considers adjacent pixel values to generate and classify map features into buildings, roads, vegetation areas, and water bodies based on predefined rules (Lobo 2018). A previous filtering process may be useful to reduce noise and stains which frequently exist in historical maps. • Corner detectors such as the Harris corner detector further enable the identification and distinction of plain surfaces, edges, and corners and therefore detect geometric features in bitmap images. One advantage of the Harris detector is its independence of scale and orientation of individual features (Collins n.d.). Further testing is required to determine individual parameters. • In decision tree-based approaches, fuzzy classifiers contribute to vectorization processes in the course of shape recognition. Based on its adjacency to others, the average belonging of a curve segment can be estimated and assigned to classes representing lines, ellipses, or curves (Liu 2004, as cited in Chen and Xie 1996). • For detecting textures of map features, pattern recognition appears to be useful. The approach of local binary pattern, for instance, enables the description of textural characteristics of a pixel’s surface in an image. By applying Gabor filters, differences between neighboring pixels and thus between textures can be detected (Prakasa 2016). Instead of performing visual analyses, similarity measures regarding the equality of objects between historical and current maps may be used for a matching process. With the help of this statistical correlation analysis, map objects are divided into elementary geometries representing nodes in a graph model. In combination with their relations to each other and depending on the relative distance, the most similar objects may be determined and assigned (Liu 2004, as cited in Li etal. 2000; Lladós etal. 2001; Peng etal. 2003). Finally, the generation of a spatial database is desirable to assign former urban images and geoobjects to current ones. In the further course of the project, a generic approach will be pursued, thus avoiding restrictions in existing solutions such as the unique applicability due to colored geometries or high-contrast contours. To further reduce limitations (see Sect.2), an initial noise reduction—originating from, e.g., hand drawings or scanning processes of historical maps—to a minimum promises an enhanced readability of historical maps and will therefore be pursued. 5.2 Conclusion With the aim of improving the comparison between historical and current maps, this project addresses a wide spectrum of challenges which users have to face in their daily working routine. The main issues have been identified with the help of a user study. Visual variables have a significant impact on the identification and differentiation of map objects. The homogenization of the diverse visual variables of maps (made by varying authors applying different production methods in different KN - Journal of Cartography and Geographic Information 1 3 eras) is supposed to improve users’ former comparing processes. Color was found to be the variable most appropriate for an intuitive detection of vegetation and water areas. Texture, however, turned out to be suited for identifying and differentiating buildings and roads. These findings ought to be investigated in further research. Compliance with ethical standards Conflict of interest The authors declare that they have no conflict of interest. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat iveco mmons .org/licen ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. References ©2018 GeoBasis-DE/BKG (©2009), Google ©2018 Microsoft Cooperation ©2018 HERE, Microsoft product screen shot reprinted with permission from Microsoft Corporation Archives nationales de Luxembourg & Musée d’Histoire de la Ville de Luxembourg. (n.d.). Mapping Luxembourg. http://www.mappi ng-luxem bourg .lu/view/index .html. Accessed 20 Sept 2018 Beconytè G (2011) Cartographic styles: criteria and parameters. In: Ruas A (ed) Proceedings of the 25th international cartographic conference (ICC 2011). Paris, France Bertin J (1973) Graphische Semiologie. Diagramme, Netze, Karten. In: Jensch G, Schade D, Scharfe W (eds) Translated from the 2nd French Edition, 2nd edn. De Gruyter, Berlin Bildungshaus Schulbuchverlage & Westermann Schroedel Diesterweg Schöningh Winklers GmbH (Eds.). (n.d.). Rom—Antike Metropole—Bauwerke. https ://www.dierc ke.de/conte nt/rom-antik e-metro pole-978-3-14-10087 0-8-141-4-2 Accessed 26 September 2018 Budig B, van Dijk TC (2015) Active learning for classifying template matches in historical maps. In: Japkowicz N, Matwin S (eds) Discovery science. 18th International conference, vol 9356. Springer, p33–47 Christophe S, Hoarau C, Boulanger L, Turbet J, Vanderhaeghe D (2015) Automatic rendering of a Cassini style. In: 27th International cartographic conference Christophe S, Duménieu B, Turbet J, Hoarau C, Mellado N, Ory J, Loi H, Masse A, Arbelot B, Vergne R etal (2016) Map style formalization: rendering techniques extension for cartography. In: Bénard P, Winnemoeller H (eds) Expressive 2016 the joint symposium on computational aesthetics and sketch-based interfaces and modeling and non-photorealistic animation and rendering. The Eurographics Association, pp59–68 Collins R (n.d.) Lecture 06: Harris corner detector (powerpoint slides). http://www.cse.psu.edu/~rtc12 /CSE48 6/lectu re06.pdf. Accessed 14 Dec 2018 Deshpande A (2016) A beginner’s guide to understanding convolutional neural networks. https ://adesh pande 3.githu b.io/A-Begin ner’s-Guide -To-Under stand ing-Convo lutio nal-Neura l-Netwo rks. Accessed 7 August 2017 Esri, DeLorme, HERE TomTom, Intermap, increment P Corp., GEBCO, USGS, FAO, NPS, NRCAN, GeoBase, IGN, Kadaster NL, Ordnance Survey, Esri Japan, METI, Esri China (Hong Kong), swisstopo, MapmyIndia and the GIS User Community Europeana (n.d.) Plan von Hamburg, 1:11 000, Lithographie, 1834 Field K (2018) Cartography, 1st edn. Esri Press, Redlands Freie und Hansestadt Hamburg, Landesbetrieb Geoinformation und Vermessung, veröffentlicht unter der Datenlizenz Deutschland— Namensnennung—version 2.0 (https ://www.govda ta.de/dl-de/ by-2-0), https ://regis try.gdi-de.org/id/de.hh/2AE6D 23E-48A54D85-BC0A-16073 7E0C8 D2 Geiling N, Esri (2013) What did San Francisco look like in the mid1800s? Smithsonian.Com. https ://www.smith sonia nmag.com/ histo ry/what-did-san-franc isco-look-mid-1800s -18094 7904. Accessed 21 September 2018 Groupe μ (1992) Traité du signe visuel. Pour une rhétorique de l’image. Paris: Seuil Hake G, Grünreich D, Meng L (2002) Kartographie. Visualisierung raum-zeitlicher Informationen, 8th edn. Berlin; New York: de Gruyter Hall M (2016a) Map of modern London inspired by John Rocque. https ://www.behan ce.net/galle ry/45230 883/Map-of-moder n-Londo n-inspi red-by-John-Rocqu es-1746-map. Accessed 19 Sep 2018 Hall M (2016b) Maps of Glasgow for Denise Mina’s “The Long Drop.” http://www.t hisi smike hall.com/maps-of-glasg ow. Accessed 19 Sep 2018 Harvard University (n.d.) Harvard Map Collection, Harvard College Library.Hamburg, drawn under the direction of Willm. Lindley, Esqr. C.E. April 1841; engraved by B.R. Davies, G6299_H3_1853_L5_3302414041 Historical (2018) In The American Heritage dictionary of the English language, 5th edn. https ://ahdic tiona ry.com/word/searc h.html?q=histo rical . Accessed 8 Oct 2018 Historical (n.d.) In Merriam-Webster. https ://www.mer r i am-webst er.com/dicti onary /histo rical . Accessed 8 Oct 2018 Huffman DP (2010) On waterlines: arguments for their Employment, Advice on their Generation. Cartogr Perspect 66:23–30 Kay S (2016) Edinburgh 2016, mapped in the style of early 1900s Ordnance Survey maps. https ://www.flick r.com/photo s/steve faeem bra/31246 13942 1/in/pool-qgis. Accessed 19 Sept 2018 Klokan Technologies GmbH (2017) Georeferencer. http://www.geore feren cer.com. Accessed 20 Sept 2018 Landesbetrieb Geoinformation und Vermessung Hamburg (2014) Hamburg einst und jetzt. Das historische Kartenbild der Innenstadt maßstabgerecht dargestellt auf dem Grundriß der Gegenwart. In: Facklam F, Fleischhauer CO (eds). Verein für Hamburgische Geschichte Larcher V, Piovan S (2018) The use of colours in historical atlases. Cartographica Int J Geogr Inf Geovis 53(2):115–128 Leyk S, Boesch R (2010) Colors of the past: color image segmentation in historical topographic maps based on homogeneity. GeoInformatica 14(1):1–21 Li C, Yang B, Xie W (2000) On-line hand-sketched graphics recognition based on attributed relation graph matching. In: Proceedings of the 3rd world congress on intelligent control and automation, IEEE, pp2549–2533 Liu W (2004) On-line graphics recognition: state-of-the-art. In: Lladós J, Kwon YB (eds) Graphics recognition. Recent advances and perspectives. 5th International Workshop, GREC 2003. Springer, p291–304 Lladós J, Martií E, José J (2001) Symbol recognition by error-tolerant subgraph matching between region adjacency graphs. In: IEEE transactions on pattern analysis and machine intelligence, vol 23. IEEE, p1137–1143 Lobo A (2018) Testing OTB segmentation methods. Talk at the OTB user day 2018, Montpellier, France. https ://gitla b.orfeo -toolb ox.org/orfeo toolb ox/otb/wikis /uploa ds/45755 e7d4f abf2b 5d461 3308f 62bd2 44/3-alobo _otb20 18.pdf. Accessed 14 Dec 2018 colored features. However, with a fluent transition between labels and other map elements of the same color their differentiation is scarcely possible within typically black and white historical maps. Due to their occasionally recurring patterns, textures are often mistakenly identified as text by automated detection processes. Tofani and Kasturi (1998), Cao and Tan (2002), Chiang and Knoblock (2014), as well as Nazari et al. (2016) defined different thresholds based on connected components to distinguish between text and other map elements. This laborious task is certainly not adaptable to a large variance of maps. Figure 2: Overlaps between labels and other map elements are supposed to be a major challenge for automated text detection. These further drawbacks do not or rarely appear within our presented map but may be a general challenge for text detection: • Wide character spacing. Cartographic labeling principles indicate a smaller spacing between characters compared to words (Chiang and Knoblock, 2014; Yu et al., 2017). According to Strabo’s specification, the horizontal space between two characters must be smaller than the largest character so that they are connected to one word (Chiang et al., 2016). This is not the case for e.g. 'Alter Wall' within the upper left part of our map subset illustrated in Fig. 1. • Extraordinarily curved labels. Strabo splits labels deviating substantially from a straight alignment into smaller parts in favor of an enhanced recognition of individual characters (Chiang et al., 2016). • Differing text sizes within a label. • Low graphical quality (Abdullah et al., 2015; Yu et al., 2017; Chiang et al., 2016). Efforts to emphasize and make use of the map’s whole RGB color range by linear contrast stretching (normalization) and global histogram equalization made only marginal improvements concerning the overall label detection rate (see (c) in Fig. 3 as well as Tab. 1). Figure 3: Detected text elements: true positives (blue) and false positives (purple). Strabo was applied to the original image subset (a), the combination of the original and the rotated input image through +45, +90, -45, and -90 degrees (b), and the combination of the original, rotated through +45, +90, -45, and -90 degrees, and enhanced (linear contrast stretching and global histogram equalization) input image (c). A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 4 of 14 Table 1. Quality of text detection by Strabo revealed by recall, precision, and f-score. The results are derived from the original, original + rotated (through +45°, +90°, -45°, -90°), as well as the original + rotated + enhanced (linear contrast stretching and global histogram equalization) input images. map (subset) number of pixels recalla precisionb f-scorec original map ĺ original + rotated map ĺ original + rotated + enhanced map as shown in Fig. 1 1081 x 881 41% ĺ 58% ĺ 66% 100% ĺ 91% ĺ 91% 58% ĺ 71% ĺ 77% subset of Fig. 1 468 x 380 34% ĺ 56% 92% ĺ 86% 50% ĺ 68% complementary map 1056 x 794 37% ĺ 70% 76% ĺ 78% 50% ĺ 74% a ݎ݈݈݁ܿܽ ൌ ୲୰୳ୣ୮୭ୱ୧୲୧୴ୣୱ ୲୰୳ୣ୮୭ୱ୧୲୧୴ୣୱା୤ୟ୪ୱୣ୬ୣ୥ୟ୲୧୴ୣୱ = percentage of correct detected text elements in respect to the total number of existing text elements b ݌ݎ݁ܿ݅ݏ݅݋݊ ൌ  ୲୰୳ୣ୮୭ୱ୧୲୧୴ୣୱ ୲୰୳ୣ୮୭ୱ୧୲୧୴ୣୱା୤ୟ୪ୱୣ୮୭ୱ୧୲୧୴ୣୱ = percentage of correct detected text elements in respect to the total number of detected elements (Pouderoux et al., 2007) c ݂െݏܿ݋ݎ݁ൌʹכ୮୰ୣୡ୧ୱ୧୭୬כ୰ୣୡୟ୪୪ ୮୰ୣୡ୧ୱ୧୭୬ା୰ୣୡୟ୪୪ with 100% indicating perfect recall and precision The algorithm developed by Chiang and Knoblock (2014) frequently generates multiple bounding boxes for individual labels which rather represent an identical one. Consequently, those bounding boxes belonging to one label overlap each other. Figure 4 illustrates how this spatial relation can be used for merging the affected bounding boxes with the aim to effectively separate off each label from the input image hereafter. Figure 4: Strabo’s outputted bounding boxes need to be merged per label to effectively separate off them from the map. In view of the aforementioned causes, overlapping bounding boxes meeting at least one of the following criteria were unified in the order as listed within an iterative procedure: 1. The overlapping area between two bounding boxes is larger than 50% of the smaller bounding box’ area (Fig. 5 (1)). 2. The distance between the centroids of two overlapping bounding boxes is larger than 1.5 times the overall average bounding box height and, at the same time, the difference between their rotation angle is less than 8 degrees (Fig. 5 (2)). To achieve the desired results, the input data was converted into a local, metric coordinate reference system before calculating each bounding box’ surface area. For criteria (1), the ratio of the overlapping area between two bounding boxes to the area of the smaller one was determined. The two considered polygons were unified into a single one for ratios of at least 50%. Preliminary testing showed that an overlap of 50% or more indicates an incorrect double detection by the algorithm and therefore an identical label. This procedure was iterated until all ratios between two bounding boxes were less than 50%. Using further Python libraries such as GeoPandas, we were able to derive the coordinates of each bounding box’ centroid. NumPy’s mean() function helped us to determine the average of the two shortest side lengths over all bounding boxes which was assumed as their initial average height. In combination with their inclination provided by Strabo and normalized to a semicircle covering 0 to 180 degrees, these two variables could be used to find cases exceeding or falling below the thresholds defined from experience for criteria (2). Again, two bounding boxes were unified as long as they fulfilled the mentioned conditions. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 5 of 14 Figure 5: Criteria contributing to a unification of two bounding boxes: overlapping area >50% of the smaller bounding box’ area (1) and centroids’ distance >1.5 times the overall average bounding box height and rotation angle difference <8° (2). As a result, each label represented on the input raster map and being localized by Strabo comprised exactly one appropriate bounding box. By applying ArcPy, Esri’s Python library for spatial data processing, the original input image could therefore be extracted by one of these bounding boxes respectively to generate individual text image areas. Being exported as individual raster files, they were rotated through their averaged rotation angle calculated on the basis of their original bounding boxes. This procedure was implemented to considerably improve the preconditions for the subsequent text recognition (Chiang et al., 2014). 3.4 Text recognition Available text recognition approaches do rarely achieve satisfactory results regarding raster maps so that additional steps become necessary (Milleville et al., 2020). With the help of a preliminary text detection an early knowledge about the exact location of text can contribute to systematically read content from input images such as historical maps. Combining text detection and recognition in an end-to-end approach not only improves recognition rates but also reduces computing time by focusing on text image areas solely (Ye and Doermann, 2015; Weinman et al., 2019). To convert the detected text image regions into a machine-readable format, resp. characters and strings, we used the free and open source engine Tesseract OCR2 (version 4.1.1) which is considered as one of the most accurate tools for optical character recognition (OCR) at present (van Strien, 2020). As all labels within the utilized map subset are in German, this language specification was defined for an improved automatic text recognition by Tesseract. Additionally, each input image should be considered as a single word. The workflow shown in Fig. A2 (see Appendix) 2 Weil et al. (2020) starts with an exemplary output from Tesseract for further processing, the string 'Fisch'. 3.5 String similarity Given a character string for each detected text image area, our aim was to roughly spatially assign them to the input map (Fig. 1). To strengthen the recognition confidence by retaining the one text string turned right way up, Chiang and Knoblock (2013) suggest a juxtaposition of recognized and suspicious characters. However, neither this methodology nor a comparison of similar strings between different maps (such as recommended by Chiang and Knoblock (2014)) considers an appropriate ground truth. In practice, OCR results are rarely precisely identical with a potential ground truth. To attain real names of streets and places, further reference values are necessary. A great variety of existing approaches (e.g. Simon et al. (2014); Weinman et al. (2019)) are based on the comparison with a regional gazetteer. This is, in some cases, available – and therefore efficiently – only for smallscale maps. We take one step further by comparing all recognized strings to an available database3 holding names of current streets and places within the region covered by our map example in Fig. 1, the city center of Hamburg. As certain street names designate e.g. historical events or circumstances and therefore are subject to only minor changes over long periods, this local geodataset could be used as a comparable similarity measure (Hanke, 2014). To effectively measure the similarity between two strings, namely an OCR output stringh indicating a historical street or place on the one hand and a list of current street names (stringc) on the other hand, their Levenshtein Distance was defined. We were able to identify street names which are likely to be identical in 3 Freie und Hansestadt Hamburg, Behörde für Wirtschaft, Verkehr und Innovation (2020) A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 6 of 14 historical and recent maps by applying two different methodologies with the help of the python library fuzzywuzzy, which implements the Levenshtein algorithm: • ratio computes the number of character edits (adding, erasing, and replacing) which have to be done to transform stringh to stringc (Yu et al., 2017) and • partial ratio computes the similarity of the shorter substring stringh within parts of the longer stringc. Here, both measures appeared to be of equal value as both individual characters might be recognized incorrectly (ĺ ratio) and only parts of strings might be identified (ĺ partial ratio). The output values are defined in percentage ranging from 0 (no similarity) to 100 (identical). Figure A2 (see Appendix) gives several output examples including their percentage value of accordance for the input stringh 'Fisch'. A low score can be an indication of either a poor OCR outcome or a great difference between the historical and current street names stringh and stringc. Additional rules being based on various own findings were defined to exclude each stringh from further processing having few (<75%) or multiple identical matching values for stringc. On the basis of a defined threshold of 75%, the street names matching those stringh were continued to be used as control points for a subsequent georeferencing. 3.6 Approximate georeferencing The dataset3 used for allocating current street names helped to perform an initial rough georeferencing of the historical map subset. Since each street within the mentioned geodataset consists of a variable number of linestrings, we defined different rules to find their centroids each representing a street’s approximate point-like location: When consisting of only one linestring, the point at half-length was assumed to represent the street’s centroid. For those streets comprising two linestrings, the interpolated point at half-length over both lines was specified as the corresponding centroid. For each street being represented by more than two lines, we built the centroid of their common rectangular bounding box. As the bottom section of Fig. A2 (see Appendix) illustrates, these labelled points served as control points for a georeferencing via affine transformation. 4 Experimental results and evaluation This section points out the results of our methodology as presented in Sect. 3. We primarily conducted tests with the map subset shown in Fig. 1 and complemented other input as necessary. 4.1 Text detection For the generation of bounding boxes each holding an individual text image area Strabo works best with RGB input images. Own tests confirmed the findings of other authors that there is no difference between lossless PNG and JPEG with smallest possible compression (at least 93% image quality (Mansurov, 2018)) using as an input data format (Milleville et al., 2020; Li et al., 2019). Our results in Tab. 1 reveal that the increase of the label detection rate was not as stark as that of Wilson (2020) when expanding an image’s spatial extent. Various challenges arose when working with Strabo. Due to their frequently similar visual characteristics, the algorithm does not differ between text and similar graphical elements such as textures or edges of map objects, particularly between those being of the same color. Suggested solutions to separate between isochromatic text and lines, such as the inclusion of connected components, may cause negative effects regarding the detection rate (Chiang and Knoblock, 2014). To facilitate further processes – in particular text recognition and string similarity – the number of detected labels could be increased by own adaptions which were already presented in Sect. 3.3. As shown in Fig. 3 (b), rotating the input image lead to a perceptible increase in the number of correctly found text elements. In reference to a ground truth, the recall could be improved from 41% regarding the original map to 58% after combining it with rotated images through +45, +90, -45, and -90 degrees respectively (Pouderoux et al., 2007). Table 1 shows that examinations with further map subsets revealed an improved recall by up to 50% through this procedure. Initial image enhancements such as linear contrast stretching and global histogram equalization could contribute once more to an improved recall of 66% when regarding Fig. 1. A slight increase of elements falsely detected as text (false positives) and therefore a decrease in the overall precision can be observed in Fig. 3 (c) as well as Tab. 1. As these did not affect the averaged accuracy measure f-score to a high degree, A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 7 of 14 we used the combined input consisting of original, rotated, and enhanced images for further processing. An accurate localization of all text areas is not necessary since the final affine transformation requires only three ground control points. 4.2 Text recognition Utilizing the derived and unified bounding boxes, the occurrence of text elements within the map could precisely be located. This enabled an improved reading of labels from the input map, the text recognition. As can be seen from Fig. A1 in Appendix, our workflow includes an extraction of all text image areas before bringing those to a horizontal orientation. Our experiences revealed that Tesseract is incapable of reading text being rotated 10 degrees and more. Recognizing rotated text is an ongoing and still not solved challenge in OCR (Ye and Doermann, 2015; Yu et al., 2017). However, map labels within the bounding boxes might be oriented in two directions. Firstly, right side up in a readable form and secondly, upside down, rotated 180 degrees. The cropped text image areas were consequently rotated through the rotation angle of their associated bounding boxes on the one hand and additional 180 degrees on the other hand. Table 2. Outputs from Strabo and Tesseract OCR as well as their Levenshtein Distance to current street names3 calculated with the help of the fuzzywuzzy library. Detected label by Strabo Rotation angle Rotated by rotation angle Recognized string by Tesseract OCR (stringh) Ground truth string from current street names3 (stringc) Average Levenshtein Distance 179° no >Speersort| Speersort 100.0% 1° no Catharıineıl Katharinenfleet 33.5% 179° no I Beichei Siebeneichen 33.5% 178° no chopenstehl Schopenstehl 98.0 % 3° no Nicola; Nieland 31.0% 167° no „ame - 0.0% yes HTollandısche Holländische Reihe 72.5% 14° no ren Wöhren, Cremon 0.0% yes ud Hude 33.5% 54° no AN - 0.0% yes | MARKT Marktweg 33.5% 55° no N - 0.0% yes Adolphs Br. Adolphsbrücke 84.0% 88° no - - - yes klopfen markt Hopfenmarkt 87.0% An appropriate input data pool for an optical character recognition by Tesseract OCR was hereby created. As the map’s original lossless PNG format performed poor for text recognition, all files were transferred in TIFF and RGB color mode. Further testing with grayscale and binary input images did not show any improvement. Regarding Tesseract’s output (examples shown in Tab. 2), a reasonable number of text strings could be A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 8 of 14 recognized distributed over the entire map. Only minor deviations from a manually prepared ground truth could be identified for horizontal labels. Similar results were also given for further tested input maps. Although Tesseract generally assumes a clean, plain input image and its model is trained on specific typefaces, interfering artifacts such as parts of lines, textures, and other map elements did not considerably deteriorate the outcomes (Rosebrock, 2018b). 4.3 Matching to current data Several concurring names could be identified between historical and current streets and places. After applying fuzzywuzzy’s (partial) ratio the previously derived centroids (see Sect. 3.3) of Tesseract’s output on the one hand and the local geodataset3 including current street names on the other hand could be matched in a satisfactory manner for our map example (Fig. 1). As seen in Tab. 2, the average Levenshtein Distance of matching strings such as Adolphsbrücke, Hopfenmarkt, Schopenstehl, or Speersort exceeded our defined threshold of 75%. We could continue to use those labels having high matching rates and a good distribution over the raster map. In combination with their centroids they served as reference points for a subsequent allocation of all remaining streets as well as for an initial rough georeferencing of the historical map. By assigning street labels to specific locations within the map, the meaning and context (semantics) of those could be specified (see Fig. 6). Figure 6: Current names of streets and places spatially assigned to the georeferenced historical map. &RQFOXVLRQVDQGRXWORRN This study can be understood as a proof of concept for an automated end-to-end workflow to extract labels from large-scale historical maps. Our findings that detection and recognition rates are generally low (<80% and <60% on average respectively) are broadly consistent with Weinman et al. (2019) and point out necessary improvements for machine learning approaches (Ye and Doermann, 2015). By combining tools addressing text detection, recognition, and string similarity with further adjustments we were able to not only increase the overall recognition rate but also to provide a base for useful ancillary information such as the names of streets and places. This may be considered a promising aspect of searchable and analyzable historical maps. Furthermore, a georeferencing, which is frequently lacking for historical maps, could roughly be made. For best results, those labels having highest similarity rates and an appropriate scattering over the map should be considered as reference points. A great benefit may be a resulting facilitated comparison between different maps such as between historical and current ones. We demonstrated the possibility of transferring the suggested approach to a variety of maps due to omitting individual adjustments. Nevertheless, disturbing factors such as interfering artifacts from building corners, textures, or map grids may occur and can therefore still be challenging for different maps. Further testing with additional maps might be helpful to specify and minimize the sources of disturbance more precisely. To improve the overall accuracy of the presented approach, we suggest connecting identified single words to complete map labels. This may be achieved by looking closely at the adjacency and similarity of rotation angles of detected text image areas. Also, map labels covering multiple lines should be considered. The certainty of true positives may therefore be increased for all substeps within our comprehensive approach. Future research might continue to use our results to label further map features and to assign those to their related geometries. The identification of geometries such as from streets, buildings, or waterbodies may be facilitated by a preceding elimination of all detected labels within a map. Segmenting and classifying map objects based on their different properties could support the establishment of ancillary, informative databases and therefore enable the analyzability of historical maps. With this kind of feature matching, not only further map objects might be identified but also a more intuitive comparison between historical and current maps would become possible. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 9 of 14 6 Data and software availability All research data and applications produced and applied within this publication can be found at https://doi.org/10.5281/zenodo.4721174 (Schlegel, 2021). The repository is structured following Sect. 3 of this paper. The results were generated using QGIS Desktop 3.16.0 (approximate georeferencing, Sect. 3.6), the command prompt in Windows 10 OS (Tesseract OCR, Sect. 3.4), the Linux (Ubuntu 18.04) command line via WindowsSubsystem for Linux (Strabo, Sect. 3.2), as well as several Jupyter Notebooks (additional adjustments, Sect. 3.3 and string similarity, Sect. 3.5) written in Python. These scripts are available under the GNU GPLv3 license. The workflow underlying this paper was partially reproduced by an independent reviewer during the AGILE reproducibility review and a reproducibility report was published at https://doi.org/10.17605/osf.io/anv9r. References Abdullah, A., Abo Alshamat, S. S., Bakhashwain, A. and Aslam, A.: Automatic text removal and replacement in scanned meteorological thematic maps, JCS., 11, 772–783, https://doi.org/10.3844/jcssp.2015.772.783, 2015. Cao, R. and Tan, C. L.: Text/Graphics Separation in Maps, in: Graphics Recognition, Algorithms and Applications, Proceedings of the 4th International Workshop on Graphics Recognition, Kingston, Canada, 7-8 September 2001, 167–177, https://doi.org/10.1007/3-540-45868-9_14, 2002. Chen, L.-H. and Wang, J.-Y.: A system for extracting and recognizing numeral strings on maps, in: Proceedings of the Fourth International Conference on Document Analysis and Recognition, Ulm, Germany, 18-20 August 1997, 337–341, https://doi.org/10.1109/ICDAR.1997.619867, 1997. Chiang, Y.-Y.: Unlocking textual content from historical maps – Potentials and applications, trends, and outlooks, in: Recent Trends in Image Processing and Pattern Recognition, Proceedings of the International Conference on Recent Trends in Image Processing and Pattern Recognition, Bidar, India, 1617 December 2016, 111–124, https://doi.org/10.1007/978-981-10-4859-3_11, 2017. Chiang, Y.-Y. (University of Southern California, Los Angeles, CA, USA): Personal communication, 2019. Chiang, Y.-Y. and Knoblock, C.: Strabo: A Complete System for Label Recognition in Maps, 26th International Cartographic Conference, Dresden, Germany, 25–30 August 2013, 2013. Chiang, Y.-Y. and Knoblock, C. A.: Recognizing text in raster maps, GeoInformatica., 19, 1–27, https://doi.org/10.1007/s10707-014-0203-9, 2014. Chiang, Y.-Y., Moghaddam, S., Gupta, S., Fernandes, R. and Knoblock, C. A.: From map images to geographic names, in: SIGSPATIAL '14, Proceedings of the 22nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Dallas, TX, USA, 4-7 November 2014, 581– 584, https://doi.org/10.1145/2666310.2666374, 2014. Chiang, Y.-Y., Leyk, S., Honarvar Nazari, N., Moghaddam, S. and Tan, T. X.: Assessing the impact of graphical quality on automatic text recognition in digital maps, Computers and Geosciences., 93, 21–35, https://doi.org/10.1016/j.cageo.2016.04.013, 2016. Chiang, Y.-Y., Duan, W., Leyk, S., Uhl, J. H. and Knoblock, C. A. (Eds.): Using Historical Maps in Scientific Studies. Applications, Challenges, and Best Practices, Springer, Cham, Switzerland, https://doi.org/10.1007/978-3-319-66908-3, 2020. Fletcher, L. A. and Kasturi, R. A.: Robust Algorithm for Text String Separation from Mixed Text/Graphics Images, TPAMI., 10, 910–918, https://doi.org/10.1109/34.9112, 1988. Freie und Hansestadt Hamburg, Behörde für Wirtschaft, Verkehr und Innovation: Straßenund Wegenetz Hamburg (HH-SIB), https://suche.transparenz.hamburg.de/dataset/strassenund-wegenetz-hamburg-hh-sib16?forceWeb=true, 2020. Goto, H. and Aso, H.: Extracting curved text lines using local linearity of the text line, IJDAR., 2, 111– 119, https://doi.org/10.1007/s100320050041, 1999. Hamburg, Germany, 1853 (Raster Image), Harvard Map Collection, Harvard College Library, https://maps.princeton.edu/catalog/harvard-g6299-h31853-l5. Hanke, C.: Hamburgs Straßennahmen erzählen Geschichte, 5th ed., Medien-Verlag Schubert, Hamburg, Germany, 2014. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 10 of 14 Iosifescu, I., Tsorlini, A. and Hurni, L.: Towards a comprehensive methodology for automatic vectorization of raster historical maps, e-Perimetron., 11(2), 57–76, 2016. Kohlstock, P.: Kartographie, 1st ed., Ferdinand Schöningh, Paderborn, 2004. Laumer, D., Gümgümcü, H., Heitzler, M. and Hurni, L.: A Semi-automatic Label Digitization Workflow for the Siegfried Map, in: Automatic Vectorisation of Historical Maps, Proceedings of the International Workshop on Automatic Vectorisation of Historical Maps, Budapest, Hungary, 13 March 2020, 55–62, https://doi.org/10.21862/avhm2020.07, 2020. Li, Z., Chiang, Y.-Y., Banisetti, S. and Kejriwal, L.: strabo-text-recognition-deep-learning (Version 0.67), GitHub repository, https://github.com/spatialcomputing/strabo-text-recognition-deep-learning, 2019. Lin, H. and Chiang, Y.-Y.: SRC: Automatic Extraction of Phrase-Level Map Labels from Historical Maps, SIGSPATIAL Special., 9(3), 14–15, https://doi.org/10.1145/3178392.3178400, 2018. Mansurov, N.: JPEG Compression Levels in Photoshop and Lightroom: https://photographylife.com/jpegcompression-levels-in-photoshop-and-lightroom, last access: 12 April 2021, 2018. Milleville, K., Verstockt, S. and van de Weghe, N.: Improving Toponym Recognition Accuracy of Historical Topographic Maps, in: Automatic Vectorisation of Historical Maps, Proceedings of the International Workshop on Automatic Vectorisation of Historical Maps, Budapest, Hungary, 13 March 2020, 63–72, https://doi.org/10.21862/avhm2020.01, 2020. Nazari, H. N., Tan, T. and Chiang, Y.-Y.: Integrating text recognition for overlapping text detection in maps, Document Recognition and Retrieval., XXIII, 1–8, https://doi.org/10.2352/ISSN.24701173.2016.17.DRR-061, 2016. Pouderoux, J., Gonzato, J., Pereira, A. and Guitton, P.: Toponym Recognition in Scanned Color Topographic Maps, in: Proceedings of the Ninth International Conference on Document Analysis and Recognition, Curitiba, Brazil, 23–26 September 2007, 531–535, https://doi.org/10.1109/ICDAR.2007.4378766, 2007. Rosebrock, A.: OpenCV Text Detection (EAST text detector): https://www.pyimagesearch.com/2018/08/20/opencvtext-detection-east-text-detector, last access: 27 October 2020, 2018a. Rosebrock, A.: OpenCV OCR and text recognition with Tesseract: https://www.pyimagesearch.com/2018/09/17/opencvocr-and-text-recognition-with-tesseract, last access: 13 November 2020, 2018b. Schlegel, I.: Empirical Study for a Deployment of a Methodology for Improving the Comparability Between Historical and Current Maps, KN – Journal of Cartography and Geographic Information., 69(2), 121– 130, https://doi.org/10.1007/s42489-019-00016-0, 2019. Schlegel, I.: Label Extraction: v1.1.0 (Version v1.1.0), Zenodo, https://doi.org/10.5281/zenodo.4721174, 2021. Simon, R., Pilgerstorfer, P., Isaksen, L. and Barker, E.: Towards semi-automatic annotation of toponyms on old maps, e-Perimetron., 9(3), 105–112, 2014. Tessdoc: Tesseract documentation. Improving the quality of the output: https://tesseractocr.github.io/tessdoc/ImproveQuality last access: 3 November 2020. Tofani, P. and Kasturi, R. (1998): Segmentation of text from color map images, in: Proceedings of the Fourteenth International Conference on Pattern Recognition, Brisbane, Australia, 16–20 August 1998, 945–947, https://doi.org/10.1109/ICPR.1998.711391, 1998. Tombre, K., Tabbone, S., Pélissier, L., Lamiroy, B. and Dosch, P: Text/Graphics Separation Revisited, in: Document Analysis Systems V., Proceedings of the 5th International Workshop on Document Analysis Systems, Princeton, NJ, USA, 19–21 August 2002, 200–211, https://doi.org/10.1007/3-540-45869-7_24, 2002. van Strien, D.: Living with Machines OCR hack: http://livingwithmachines.ac.uk/living-with-machinesocr-hack, last access: 25 August 2020. Velázquez, A. and Levachkine, S.: Text/Graphics Separation and Recognition in Raster-Scanned Color Cartographic Maps, in: Graphics Recognition, Recent Advances and Perspectives, Proceedings of the 5th International Workshop on Graphics Recognition, Barcelona, Spain, 30–31 July 2003, 63–74, https://doi.org/10.1007/978-3-540-25977-0_6, 2004. Weil, S. et al.: Tesseract Open Source OCR Engine (main repository) (Version 4.1.1), GitHub repository, https://github.com/tesseract-ocr/tesseract, 2020. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 11 of 14 Weinman, J. (2013): Toponym recognition in historical maps by gazetteer alignment, in: Proceedings of the 12th International Conference on Document Analysis and Recognition, Washington, DC, USA, 25–28 August 2013, 1044–1048, https://doi.org/10.1109/ICDAR.2013.209, 2013. Weinman, J., Chen, Z., Gafford, B., Gifford, N., Lamsal, A. and Niehus-Staab, L.: Deep neural networks for text detection and recognition in historical maps, in: Proceedings of the 15th International Conference on Document Analysis and Recognition, Sydney, Australia, 20–25 September 2019, 902–909, https://doi.org/10.1109/ICDAR.2019.00149, 2019. Wilson, D.: Finding words in maps: http://livingwithmachines.ac.uk/finding-words-inmaps, last access: 25 August 2020. Ye, Q. and Doermann, D.: Text Detection and Recognition in Imagery: A Survey, TPAMI., 37, 1480– 1500, https://doi.org/10.1109/TPAMI.2014.2366765, 2015. Yu, R., Luo, Z. and Chiang, Y.-Y. (2016): Recognizing text in historical maps using maps from multiple time periods, in: Proceedings of the 23rd International Conference on Pattern Recognition, Cancun, Mexico, 4–8 December 2016, 3993–3998, https://doi.org/10.1109/ICPR.2016.7900258, 2017. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 12 of 14 Appendix Figure A1: Workflow from label detection to recognition for a map subset including interposed further adjustments. A GILE: GIScience Series, 2, 12, 2021 | https://doi.org/10.5194/agile-giss-2-12-2021 13 of 14 KN - Journal of Cartography and Geographic Information 1 3 2.3 Vectorization andVector Enhancement As vector data can be better processed and analyzed than raster data, a majority of the mentioned authors proceed with a vectorization of extracted map objects. Brown (2002) and Arteaga (2013) use specific software tools to, respectively, vectorize the outlines of geologic structures and buildings from historical maps. Vectorization tools are also provided within ArcGIS, GRASS GIS (Gede etal. 2020), and the GDAL library (Jiao etal. 2020). To purge vectorized objects, further simplification processes may follow. Multiple software and tools, including eCognition, QGIS, ArcGIS (Godfrey and Eveleth 2015), SAGA GIS (Gede etal. 2020), R (Arteaga 2013), and Python libraries, implement pre-built functions to smooth or simplify lines or polygons and to remove outliers, spikes, and other artifacts. 2.4 Object Matching For the direct comparison of vector objects from different maps from various times, distance and similarity measures may be promising (Xavier etal. 2016). Matching geometries between different inputs is frequently performed on the basis of shape or spatial similarities (Tang etal. 2008) or identical attribute values (Frank and Ester 2006). However, semantic similarity approaches are not feasible as scanned historical maps usually hold no ancillary information. Even if names of roads or buildings were available—e.g., by a preceding text recognition—they would need to be assigned to their corresponding geometries. When analyzing geometric relations, such as overlapping areas or distance measures (e.g., Euclidian or Hausdorff distance), only relative distances between objects are considered. This technique is useless when comparing not yet georeferenced datasets (Xavier etal. 2016). Region-based shape descriptors (e.g., area, convex hull, Moment or Grid descriptor, see Ahmad etal. (2014)) regard all pixels within a shape and may therefore be promising for a comparison between identical real-world objects from different inputs. But due to uncertainties, they are rather considered complementary matching approaches. Additional similarity measures are necessary (Xavier etal. 2016). By regarding the spatial relationship between objects, Stefanidis etal. (2002) quantified their distances and relative positions. Samal etal. (2004) and Kim etal. (2010) consulted third objects to create an overall geographic context. Also, Sun etal. (2021) regarded spatial relationships by linking identical real-world objects from different historical maps. However, their knowledge graph approach presupposes the existence and assignment of labels to their corresponding geometries. 3 Data For a proof of concept of our suggested methodologies, a large-scale (~ 1:11,000) historical map from the middle of the nineteenth century was chosen, which has already been object of research within related studies (Schlegel 2019, 2021). The original non-georeferenced and undistorted version of the map scan was cropped to a smaller extent (~ 1000 × 800m in reality) for reasons of runtime compression within all processes. No map projection is known. The map subset in Fig.1 shows the city center of Hamburg with blocks of buildings, roads, and water areas. Apart from subsequently colorized water areas, the map is drawn in black and white. Many data suppliers provide their raster scans with a resolution of 300 ppi which is considered adequate for object extraction purposes (Pearson etal. 2013). Lower pixel densities induce blurring and pixelation, whereas higher values tend to highlight interfering artifacts from, e.g., folds in paper, discolorations, or smudges (Peller 2018). We continued to work with the TIFF format (without compression) as it is lossless concerning the image’s original pixel values (Gede etal. 2020). To demonstrate the transferability of the workflow, two more large-scale historical maps covering the same spatial area were used in the further course (see Fig.9a, b). They all differ in their visual appearance and complexity in terms of contrasts, textures, or the existence of labels and gridlines. For comparing the described data to a current counterpart, official vector datasets including recent polygonal buildings (Landesbetrieb Geoinformation und Vermessung 2022) and line-type roads (Behörde für Verkehr und Mobilitätswende (BVM) 2020) were used. 4 Object Extraction 4.1 Preparation fortheElimination ofLabels As similar color values and overlaps between labels and other map objects impede a clear discriminability, an initial elimination of labels designating real-world objects significantly contributes to a facilitation of object recognition processes. We suggest to make use of the output from previous label detection attempts (see Schlegel (2021)): vector bounding boxes comprising text image areas, which can be seen in Fig.1. An exemplary text image area is shown in Fig.2a. With the aim to eliminate its content from the map, it was initially cropped by means of its original bounding box (see Fig.2b) and rotated to the horizontal by its angle of alignment (Fig.2c)—calculated by the used text detection tool Strabo (Li etal. 2018; Chiang and Knoblock 2014). However, these text image areas do not only include characters, KN - Journal of Cartography and Geographic Information 1 3 but also edges of buildings, which is an outgrowth of Strabo (see upper margin in Fig.2c). This is counterproductive within the subsequent step of building segmentation as these image areas were supposed to be entirely eliminated from the map. Thus, building edges would become distorted. To retain these important edges, all pixels within a bounding Fig. 1 Map subset showing the city center of Hamburg (Harvard Map Collection, Harvard College Library etal. (n.d.)) with bounding boxes containing labels produced by a previous text detection Fig. 2 Steps for separating building edges from labels shown with an exemplary dataset: a input map with bounding box containing text image area, which then was b cropped, c aligned horizontally, and d converted into a binary as well as e athree-class mask. The f resulting bounding box excluding building edges was g turned back to its original orientation KN - Journal of Cartography and Geographic Information 1 3 box were differentiated by text and parts of buildings. A user-defined thresholding helped to generate a binary mask consisting of dark “foreground” and bright “background” pixels (see Fig.2d). A further “foreground” differentiation was needed to separate building edges from text pixels. However, similar color values, overlaps, and smooth transitions between text and buildings were challenging. For reclassifying former “foreground” into either “text” or “building edge” pixels, multiple thresholds and conditions had to be applied (Fig.2e). As labels designating roads most commonly run parallel to nearby building edges, this step was performed row-wise. As Fig.2f indicates, all pixels representing “text” and “background” were combined and vectorized. The resulting polygonal bounding box was turned back by its initial rotation angle (see Fig.2g) and then used within the following object extraction steps. 4.2 Object Detection andRecognition To detect homogeneous image regions and extract objects such as buildings or water areas from large-scale historical maps, we used object-based image analysis. In contrast to pixel-based approaches (e.g., Maximum Likelihood, Clustering, or Thresholding), which only regard spectral differences between pixels, OBIA generates image objects also based on common textures, shapes, context, etc. and is, therefore, more suitable for historical maps with limited spectral information and heterogeneous appearances (Blaschke etal. 2014; Hussain etal. 2013). As none of the many free and open source packages available for semi-automated feature extraction produces comparable results, we made use of the proprietary software eCognition Developer 10.2 to generate GIS compatible data from a historical map via OBIA (Kaur and Kaur 2014). eCognition converts user-defined rule sets—built-up from functions, filters, statistics, etc. for image segmentation and classification—into machine-readable code. These concatenations of algorithms can be easily transferred to other images (Trimble Inc. 2022). As Fig.3a indicates, a first rough differentiation between dark (foreground) map features (e.g., buildings and labels) and the map’s bright background (water areas, roads, and places) was enabled by thresholding the input TIFF. The content of the labels’ bounding boxes, as shown in Fig.2f, was simply classified as “background” and could therefore be eliminated (see Fig.3b). The detection of further map objects is therefore significantly facilitated on the one hand and building edges remain unaltered on the other hand. To extract contours of buildings, an edge detector was applied to the image. The building texture’s repeating pattern could be detected by means of a gray-level co-occurrence matrix—which measures the vertical invariance of adjacent pixel pairs—and analyzed by texture descriptors (Chaves 2021; Trimble Inc. 2021). Regarding the original map in Fig.1, public buildings (e.g., the townhall or churches) have a significantly darker texture and could, therefore, clearly be differentiated from other buildings based on their gray values. Water areas were identified by thresholding the RGB blue channel as well as applying supplementary texture descriptors to avoid false positives. 4.3 Vectorization Generally, OBIA results in raster files containing individual image objects, subdivided into predefined single classes. For further processing and analysis purposes, a vectorization of this data is inevitable. Based on experiences of Iosifescu etal. (2016) and Arteaga (2013), we applied GDAL’s polygonize function to perform a raster-to-vector conversion Fig. 3 Foreground objects separated from the map’s background a before and b after eliminating labels KN - Journal of Cartography and Geographic Information 1 3 for each map class. Several functions to simplify and smooth the vectorized map features, to close inlying minor gaps, and eliminate small isolated polygons were compiled within an end-to-end Python script. This way, interfering artifacts (e.g., islands, protrusions, or spikes) stemming from an imprecise segmentation or undetected labels could be handled. The resulting polygons representing (public) buildings and water areas are shown in Fig.4 and can be processed within future analysis operations. 5 Linking Historical andCurrent Datasets Compared to previous studies dealing with object extraction from historical maps, we go one step further and present an exemplary way of how qualitative and quantitative evaluations of long-term changes within a cityscape may be practically enabled. We therefore spatially assigned a more recent vector dataset to the historical counterpart as shown in Fig.7. Our aim was to automate this coarse georeferencing process as far as possible. Due to changing names of roads and buildings over time, the lack of indepth information, or simply imprecise scales, distances, and directions within historical maps, we used the previously extracted geometries for georeferencing purposes (Rumsey and Williams 2002). As can be seen from Fig.5, churches and other municipal buildings still exist over time and, beyond that, do not substantially change their basic shape and geographic location over time. Therefore, their object shapes could be matched and used for the definition of control points in the further course of georeferencing (Skopyk 2021; Havlicek and Cajthaml 2014). 5.1 Shape Matching To define matching georeferencing control points between the historical and current dataset, identical real-world objects are to be identified. We, therefore, measured the shape similarity between the extracted public buildings shown in Fig.5 (Sun etal. 2021; MacEachren 1985). A matching based on Fig. 4 Vectorized and revised features of the historical map KN - Journal of Cartography and Geographic Information 1 3 spatial or semantic (attribute-based) similarities was impractical due to the lack of a coordinate system as well as further information concerning the historical map. As Fig.5 indicates, a side-by-side comparison between geometries of public buildings extracted from the historical map on the one hand and the official vector dataset containing current buildings on the other hand was performed. We implemented a matching of their shapes based on their Intersection over Union (IoU). After adjusting the aspect ratios of corresponding counterparts via rectangular bounding boxes (“envelopes” (Esri 2022)), their respective deviations could be quantified via IoU. As can be seen from Fig.6, a building geometry and its envelope together form a binary mask—consisting of the values 1 (building geometry) and 0 (envelope). A final superimposition of these masks helped to determine their overlapping area (intersection) proportionally to their common area (union) (see Fig.6). All “building” pixels with a value of 1 were considered for the IoU calculation, which was conducted with the help of Python’s numpy library. Table1 summarizes the IoU results for all detected public buildings continued to use for georeferencing purposes. 5.2 Georeferencing 5.2.1 Method Overview The centroids of those geometries with the closest matches (see highlighted cells in Table1) were defined as control points for a semi-automated, rough georeferencing between the historical and current dataset. To preserve the objects’ shapes and to keep spatial deformations to a minimum within the historical data, an affine transformation of all current buildings and roads was conducted. This Fig. 5 Vectorized public buildings (churches and townhall) from the historical map (upper row) and their counterparts from the current dataset (bottom row) Fig. 6 Intersection over Union between the historical St. Petri Kirche and a its current counterpart as well as b the current St. Katharinen Kirche. The aspect ratio of the geometries’ envelopes was adjusted to one another, respectively KN - Journal of Cartography and Geographic Information 1 3 was done via QGIS Vector Bender (Dalang 2019) using the three matching object pairs highlighted in Table1 as well as Fig.7. In our test case, only three control points with sufficient pointing accuracy could be found—such a small number is quite typical for historical maps. However, if available, a larger quantity of control points is advisable Table 1 Numerical results from Intersection over Union between historical and current buildings’ geometries Historical map St. Katharinen Kirche St. Petri Kirche Rathaus Current dataset St. Katharinen Kirche 83,9% 74,9% 45,0% St. Petri Kirche 79,3% 84,4% 43,1% Rathaus 47,5% 45,9% 58,5% St. Jacobi Kirche 82,1% 80,2% 42,6% Mahnmal St. Nikolaia54,0% 49,8% 26,1% aThe St. Jacobi Kirche was not classified as public building by eCognition whereas the (Mahnmal) St. Nikolai was reconstructed in another city district after being mainly destroyed during World War II and leaving only its tower until today (Claussen n.d.). a The St. Jacobi Kirche was not classified as public building by eCognition whereas the (Mahnmal) St. Nikolai was reconstructed in another city district after being mainly destroyed during World War II and leaving only its tower until today (Claussen n.d.) Fig. 7 Georeferenced current buildings and roads based on the centroids of the highlighted St.-Katharinen-Kirche, St.-Petri-Kirche, and Rathaus KN - Journal of Cartography and Geographic Information 1 3 to benefit from over-determination for the transformation process. Figure7 shows that a georeferencing between historical and current geodata gives the chance to directly compare their contents and, thus, to evaluate changes within an area over time (Iosifescu etal. 2016). 5.2.2 Error Estimation For a minimum of transformation bias, it is generally recommended to evenly spread georeferencing control points throughout the input (Clark and MacFadyen 2020). We, therefore, evaluated affine transformation results using alternative control points. In Table2, the resulting offsets between the two datasets are quantified for three different cases. Case a) represents the initial georeferencing with centroids of three public buildings used as control points, as illustrated in Fig.7. In case b), the upper right centroid (St.-Petri-Kirche) was replaced by the one of another public building located rather at the edge of the input (St.-JacobiKirche, see right margin in Fig.7), whereas in case c), distinct crossroads close to the map’s edges provided an optimum distribution of control points. Due to missing control points in the lower left image area, cases a) and b) resulted in greater deviations compared to c) (see Table2). However, a visual inspection revealed only minor differences between the three approaches. In view of our objective, which was to roughly locate current map features and to enable a visual comparison of these with their historical counterparts, all three approaches delivered satisfactory results. 5.3 Geographic Context To further assess the quality of the chosen control points, their geographic context was regarded. Based on the model from Samal etal. (2004), a contextual similarity between real-world matching map features was exemplarily computed for case a). Figure8 shows an example of how a proximity graph—connecting the centroids of four building geometries with the one of a known geometry from 5.1—was built for each dataset. The offset between the two overlaying datasets could then be expressed by the length and angle of displacement between corresponding centroids (see Table3). The largest deviations of up to 1.8cm (72m in reality) and 3.4cm (~ 34m) on average between Table 2 Average of absolute offset, corresponding standard deviation, and RMSE between current, spatially assigned and historical reference data The deviations were calculated based on the absolute distances between unambiguously matching crossroads. Unit: meters Mean offset Standard deviation Root mean square error Case a 359 439 37 Case b 275 386 29 Case c 180 278 17 Fig. 8 Exemplary proximity graph with displacement vectors between historical and current data Table 3 Absolute offsets between historical and georeferenced current data, expressed by the displacement vectors’ length and angle shown in Fig.8. The real lengths were derived from the historical map’s original scale (Hamburg Feet) Central building Vector # Length Angle (°) Reality (m) Map (cm) St. Katharinen 1 72.0 1.8 −56.6 2 11.6 0.3 −105.8 3 24.7 0.6 −7.6 4 30.6 0.7 16.1 KN - Journal of Cartography and Geographic Information 1 3 historical and georeferenced current dataset deemed to be reasonable in view of our application case. 6 Applicability oftheObject Extraction Workflow The following sections demonstrate in an exemplary way the transferability of the object extraction workflow described in 4.2 by means of two more historical map subsets illustrating the same spatial extent of the city of Hamburg (hereinafter named as “map A” and “map B”, see Fig.9a and b, respectively). Minor changes had to be conducted to achieve optimum results for the two different maps. 6.1 Map A Rough building structures could be identified when applying the OBIA workflow to the label-less map A, illustrated in Fig.9a. However, surface-filling geometries were not detected so that single processing steps had to be modified and added. For instance, to detect the conspicuous hatching of building geometries, a simple line detection algorithm was implemented. Water areas could be identified based on their outstanding hatching pattern consisting of isolated dashes. Small gaps were filled and object contours were closed with the help of morphological closing, which avoids expanding the segmented objects (Chiang etal. 2014; Gede etal. 2020). The resulting classified image objects can be seen in Fig.10a. Fig. 9 Alternative input maps by a Sammlung Christian Terstegge (n.d.) and b Harvard Map Collection, Harvard College Library etal. (2008) Fig. 10 Segmentation results by the use of the adjusted workflows for a map A and b map B KN - Journal of Cartography and Geographic Information 1 3 6.2 Map B Due to its monotonous appearance, a straightforward applicability of the workflow described in Sect.4.2 was not feasible for map B. The dark contours of building objects were extracted by means of thresholding so that their enclosed textures could simply be classified as buildings as well. Also, water areas could be identified by regarding their distinctive texture. Labels were differentiated and classified based on their neighborly relations to buildings and the maps’ background (roads and places), respectively. As can be seen from Fig.10b, these relations were not unambiguous in each case. The map objects’ quality highly depends on the map’s complexity. With a greater degree of complexity, OBIA results became less satisfying. Apart from visual overload, further challenges may impede a segmentation of historical maps: • Stains, folds, and tears in the maps’ original material, • detailed map objects and symbols (e.g., roofs of buildings, trees, or blades of grass), • heterogeneous or absent textures, or • overlaps between labels and other map features. 7 Potential Future Applications With the geometries resulting from the workflow described in chapters4 and 5, valuable analysis and comparison processes concerning urban morphological developments become possible. According to a preceding user study (Schlegel 2019), comparisons between historical and current maps mainly relate to buildings and roads as well as general transformations in the urban structure. Figure11 shows two potential use cases: Users may select a historical building whilst, in the background, an intersection algorithm finds appropriate current buildings and outputs related information such as its name or area (see Fig.11a). Alternatively, current road names might be queried. By selecting a historical road section, the current road name may be returned from a database using the intersection area between the bounding box of the former and the corresponding line feature of the latter (see Fig.11b). Fig. 11 Exemplary use cases: comparing a historical with a a current building and b a current road section by selecting a historical feature. Based on their intersection area, related information is returned from a corresponding database KN - Journal of Cartography and Geographic Information 1 3 8 Discussion For a considerable enhancement of object extraction processes from historical maps, a preceding elimination of labels is advantageous. Based on the results of a text detection tool used in the course of previous research (Schlegel 2021) and assuming that labels run parallel to roads and edges of buildings, we were able to eliminate straight text. As gray values of labels often do not differ significantly from the ones of adjacent map objects such as contours or textures of buildings, their existence complicates efforts towards the extraction of geometries from a historical map. In the present work, text was separated from features of similar color by thresholding techniques so that a more precise object extraction became feasible. This procedure is irrespective of any preceding map enhancements or georeferencing and efficient especially for monochrome historical maps having a heterogeneous background. In contrast to other studies, our suggested approach neither mistakenly removes other map features nor substitutes original pixel values. Nevertheless, an optimization of the preceding text detection should be undertaken so that all map labels are considered for elimination in future research. A main purpose of this study was to pave the way for a straightforward comparison of large-scale historical maps with recent counterparts. Vectorized and georeferenced map features allow their analyzability and searchability in the further course. With the help of enhanced objectbased image analysis as well as subsequent vector refinement and linking processes, we address this issue within a semi-automated workflow. By applying OBIA approaches, not only spectral, but also textural, shape-dependent, or contextual characteristics of map objects are considered for their identification. Available techniques from image and vector processing contributed to an adequate quality of extracted features and to make a large-scale historical map analyzable and comparable. This is inevitable for investigating urban transformations over time. On the downside, specific software, knowledge, and, in some instances, subjective and individual solutions are required, especially within the object extraction domain. Consequently, a fully automated workflow is not realizable. A critical view on the results shows that these strongly depend on a maps’ complexity and the quality of the underlying bitmap. All processing steps applied to a bitmap are affected by its color depth, format, and resolution (Gede etal. 2020). Further improvements of the suggested methodology may involve a consideration of additional maps, e.g., showing other cities, and algorithms. By roughly georeferencing large-scale historical with current maps and putting these on top of one another, a direct comparison of their individual objects is facilitated. We suggest to define georeferencing control points based on the shape and context similarities of map features such as public buildings. It is assumed that these buildings were already classified as such from preceding object extraction. To measure the similarity of objects between a historical and a current dataset, two methodologies are presented. It was found that corresponding geometries of churches have a high similarity as, usually, their shapes remain unchanged over centuries. When comparing these for matching purposes, both maps need to have a similar scale and degree of generalization. However, shape similarities are not invariably unambiguous. Distortions induced by adjusting the geometries’ aspect ratios may lead to biased results. Further similarity measures, such as the geographic context, are therefore necessary. The consideration of the geographic context of objects proved beneficial for a quality control of the control points defined for the final georeferencing. Its outcomes depend on the subjective choice of reference points. Finally, georeferencing current with historical maps does not necessarily improve their accuracy. Depending on the particular application, it must be noted that shapes and lines, distances, or proportions may be distorted. Regarding our objective of comparing historical map content, the presented rough georeferencing proved to be satisfying for potential future applications. 9 Conclusion andOutlook A major purpose of this study was to present the feasibility of a holistic workflow. Within an end-to-end solution, semi-automated approaches to extract and vectorize features from a large-scale, mainly monochrome, historical map were developed and applied for the purpose of providing knowledge, of analyzability (e.g., in GIS), and comparability. A concluding georeferencing enabled a straightforward comparison to current counterparts and, in the further course, to understand changes within a cityscape over time. A significant contribution could be made by previously eliminating map labels and separating them from other objects to improve their extraction results. It was shown that a rough georeferencing is sufficient for the juxtaposition of historical and current map objects. With the presented methodologies, an appropriate spatial allocation without distorting map objects was achieved. The present findings confirm that each map has an individual need for adaption, but only minor adjustments are required to apply the suggested approaches to maps having a similar degree of complexity. Overall, our results make a major contribution to extract valuable information from large-scale historical maps by