Repositorio Institucional de Documentos

2012 22 Aneta Jadwiga Florczyk Search improvement within the geospatial web in the context of spatial data infrastructures Departamento Director/es Informática e Ingeniería de Sistemas Zarazaga Soria, Francisco Javier López Pellicer, Francisco Javier Director/es Tesis Doctoral Autor Repositorio de la Universidad de Zaragoza – Zaguan http://zaguan.unizar.es UNIVERSIDAD DE ZARAGOZA Departamento Director/es Aneta Jadwiga Florczyk SEARCH IMPROVEMENT WITHIN THE GEOSPATIAL WEB IN THE CONTEXT OF SPATIAL DATA INFRASTRUCTURES Director/es Informática e Ingeniería de Sistemas Zarazaga Soria, Francisco Javier López Pellicer, Francisco Javier Tesis Doctoral Autor 2012 Repositorio de la Universidad de Zaragoza – Zaguan http://zaguan.unizar.es UNIVERSIDAD DE ZARAGOZA Departamento Director/es Director/es Tesis Doctoral Autor Repositorio de la Universidad de Zaragoza – Zaguan http://zaguan.unizar.es UNIVERSIDAD DE ZARAGOZA SEARCH IMPROVEMENT WITHIN THE GEOSPATIAL WEB IN THE CONTEXT OF SPATIAL DATA INFRASTRUCTURES Aneta Jadwiga Florczyk PhD DISSERTATION RESEARCH ADVISORS Dr. Francisco Javier Zarazaga–Soria Dr. Francisco Javier López–Pellicer May 2012 Computer Science and Systems Engineering Department Universidad de Zaragoza ©Copyright by author, copyrightyear All Rights Reserved iii Acknowledgments I would like to thank all members of the IAAA research group of the University of Zaragoza, to which I had pleasue belong during the thesis development, for their collaboration and support in any aspect, and especially, my research advisors, F.Javier Zarazaga–Soria and F.Javier López–Pellicer for their endless forbearance and patience. Also, I would like to express my gratitude to members of the IFGI of the University of Muenster, Germany, and every other person I have met there, who made effort to make my stay a wonderful experience, which has contributed to my profesional and personal development. I feel specially grateful to proffesor Werner Kuhn, Tomi Kauppinen and Patrick Maué for their help in this development. I would like to thank the people that have reviewed this thesis. Despite all of their help, I take full responsibility for any errors or omission herein. Finally, I would like to thank particularly my parents for their unconditional support during the work on this thesis. They always managed to find the right words to encourage me to follow the path I have chosen, even if this meant personal sacrifices for them. v List of Figures 1.1 Spatial dimension of information in information management. . . . . . . . . . . . . . 2 1.2 Augmentation property of Geospatial Cyberinfrastructure that supports domain and cross–domainresearch.................................... 4 1.3 Geospatial Portal Reference Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Geospatial Portal Reference Architecture Services Distribution. . . . . . . . . . . . . 6 1.5 Geoportals as access points into multidisciplinary GCI cube for exploring research results............................................. 9 1.6 Scenario for searching for information about a feature within various resources, according to a general approach and its enhanced version. . . . . . . . . . . . . . . . . 12 1.7 Scenario for searching within an SDI for resources that provide information about a feature, according to a general approach and its enhanced version. . . . . . . . . . . 12 1.8 Scenario for searching for resources among different SDIs. . . . . . . . . . . . . . . . 13 1.9 Nested evaluation frameworks of an IR system. . . . . . . . . . . . . . . . . . . . . . 15 2.1 Direct and indirect geospatial representation. . . . . . . . . . . . . . . . . . . . . . . 21 2.2 Extracting Quality Factors of Web Service. . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 Structure of Web Services quality factor. . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4 Different approaches to geographic information quality from the quality management viewpoint........................................... 27 2.5 The generalised abstraction of the geocoding process. . . . . . . . . . . . . . . . . . . 28 2.6 The alternative paths in geocode production. . . . . . . . . . . . . . . . . . . . . . . 29 2.7 The selected characteristics of geocoding service. . . . . . . . . . . . . . . . . . . . . 30 2.8 Workflow for the metadata generation of a mediator service. . . . . . . . . . . . . . . 32 2.9 Overview of the Compound Geocoding Service Architecture. . . . . . . . . . . . . . . 34 2.10 Domain data model evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.11 Mapping domain data model to data models of Google and IDEZarSG services. . . . 37 2.12 Mapping domain data model to data models of CartoCiudad and Cadastre services. 38 2.13 Application of the domain data model. . . . . . . . . . . . . . . . . . . . . . . . . . . 40 xiii 2.14 Layer view of the compound geocoding service and client components. . . . . . . . . 40 2.15 Introduction and publication of urban incidents. . . . . . . . . . . . . . . . . . . . . 41 3.1 Modelling the spatial representation of a geo–concept. . . . . . . . . . . . . . . . . . 44 3.2 Overlapping MBBOXES of administrative areas (a case from Spain). . . . . . . . . . 49 3.3 Geoidentifiers and modelling the spatial representation of a geo–concept. . . . . . . . 50 3.4 Administrative geography as support for services catalog. . . . . . . . . . . . . . . . 53 3.5 Relations between the service description and the administrative geography. . . . . . 53 3.6 Overview of the method for the orthoimage layer detection. . . . . . . . . . . . . . . 59 3.7 Workflow of the description–based analysis. . . . . . . . . . . . . . . . . . . . . . . . 61 3.8 Workflow of the content–based analysis. . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.9 Example of image division in the collection procedure. ................. 64 3.10 Pixel test heuristic...................................... 67 3.11 Class diagram of the implemented WPS. . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.12 Sequence diagram of the communication between a client and the OGC WPS for the identification of the orthoimage WMS layer. . . . . . . . . . . . . . . . . . . . . . . . 73 3.13 Services catalog as the support component for application based on on–the–fly data integration. ......................................... 75 3.14 Integration of the WPS for orthoimage detection within the Virtual Spain project. . 75 4.1 Overview of the system functionality. . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.2 Overview of the system architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.3 Overview of the coverage estimation method. . . . . . . . . . . . . . . . . . . . . . . 86 4.4 Overview of the content–based heuristic. . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.1 Three kinds of search activities and exploratory search. . . . . . . . . . . . . . . . . 104 5.2 Geospatial service taxonomy proposed in Bai et al. (2009). . . . . . . . . . . . . . . . 114 5.3 Multi–layer logical structure of the URN taxonomy. . . . . . . . . . . . . . . . . . . 115 5.4 Prototyped GUI of the search result offered by the Geospatial Web Search Engine. . 118 5.5 Overview of the searching and indexing tasks performed by Geospatial Web Search Engine. ........................................... 121 5.6 Overview of the architecture of the Geospatial Web Search Engine. . . . . . . . . . . 125 xiv Listings 2.1 Example of RDF description of a mediator service which uses a set of the Cadastre ServicesofSpain....................................... 39 3.1 Example of RDF description which represents the Zaragoza municipality. . . . . . . . 51 3.2 SPARQL request pattern for service selection via an MBBOX. . . . . . . . . . . . . 55 4.1 Metadata generated when applying the coverage estimation method. . . . . . . . . . 91 B.1 SKOS vocabulary for the OWS service Taxonomy. . . . . . . . . . . . . . . . . . . . 139 B.2 SKOS vocabulary for the OWS resources. . . . . . . . . . . . . . . . . . . . . . . . . 145 B.3 SKOS vocabulary for the domains of OWS services (an example). . . . . . . . . . . . 145 B.4 SKOS vocabulary for the providers of OWS services (an example). . . . . . . . . . . 146 C.1 Examplesofrequests. ................................... 148 C.2 Example of the GetFeature response which retrieves only one instance. . . . . . . . . 149 xv List of Algorithms 1 Algorithm for collecting image fragments of a WMS layer. . . . . . . . . . . . . . . . 65 xvii Nomenclature CI Cyberinfrastructure CSDGM Content Standard for Digital Geospatial Metadata DC Dublin Core DL Digital Library ESDIN European Spatial Data Infrastructure with a Best Practice Network FGDC Federal Geographic Data Committee GCI Geospatial Cyberinfrastructure GEOSS Global Earth Observation System of Systems GIBO Geographic information–bearing objects GIR Geographic Information Retrieval GIS Geographic Information System GML Geography Markup Language GPS Global Positioning System GUI Graphical User Interface HTML HyperText Markup Language HTTP Hypertext Transfer Protocol INSPIRE Infrastructure for Spatial Information in the European Community IR Information Retrieval ISO International Organization for the Standardization xix ISO/TC 211 ISO Technical Committee 211 Geographic information/Geomatics LBS Location based service MBBOX Minimum Bounding Box NAP North American Profile of ISO 19115: Geographic Information - Metadata NER Named Entity Recognition NSDI National Spatial Data Infrastructure OASIS The Organization for the Advancement of Structured Information Standards OGC Open Geospatial Consortium OWL Web Ontology Language OWS OGC Web Service QoS Quality of Service RDF Resource Description Framework RDFS RDF Schema SDI Spatial Data Infrastructure SDTS Spatial Data Transfer Standard SE Search engine SEIS European Shared Environmental Information Space SKOS Simple Knowledge Organization System SLA Service–level agreement SOA Service Oriented Architecture ToS Terms of Service UDDI Universal Description, Discovery and Integration UNGIWG Unated Nations Geographic Information Working Group UNSDI United Nations Spatial Data Infrastructure URI Universal Resource Identifier xx WSDL Web Service Definition Language WSQF OASIS Web Service Quality Factor WSQM OASIS Web Services Quality Model xxi xxii 1.2. MOTIVATION 7 A search for specific information is an important issue in any information system. In the SDI context, catalogue services8are discovery and access systems for geographic information that use indexed and searchable metadata against which intelligent geospatial searches can be performed within or across SDI communities (Nebert, 2004). The metadata profiles applied in SDIs use different standards as their base. For example INSPIRE Implementing Rules on metadata (INSPIRE DTM and EC/JRC, 2010) identifies ISO 19115:2003 (ISO, 2003), and FGDC recommends Content Standard for Digital Geospatial Metadata (CSDGM) (FGDC, 1998a) and North American Profile (NAP) of ISO 19115:2003 (ANSI, 2009). Although metadata standards vary across SDIs, crosswalks enable the translation of this information in order to make it conform to a selected metadata standard or profile (Nogueras-Iso et al., 2004; Batcheller, 2008; Khoo and Hall, 2010). For example, Nogueras-Iso et al. (2004) presents a transformation between ISO 19115 Core and Dublin Core (DC) metadata standards. The DC metadata standard is the outcome of an open metadata initiative for the description of cross–domain information resources (Powell et al., 2007) that has become an ISO Standard (ISO Standard 15836:2009 (ISO, 2009)). Crosswalks enable users to perform different search strategies among different SDIs, such as federated or centralised search. For example, Leite et al. (2006) propose a Web–based GIS architecture where the central point is a catalogue that offers federated search over distributed catalogues. Another proposal from the geospatial community is a multi–catalogue search engine that searches across various catalogues that implement the OGC Catalogue Service standard (Nebert et al., 2007) and offers an integrated result (Li et al., 2011). Traditionally, Web resources, from Web pages to SDI geoportals, have not received much attention from the geospatial community. Metadata about these resources, often potentially interesting for users and stakeholders, is seldom found in SDI catalog services. However, indices point out that the status quo may change. The number of geoportals available online is growing, a fact that paradoxically increases the difficulties of searching for geospatial information. Geoportals are the visible part of a GCI that can be exploited by a potential user, which will try to discover such areas of interest (i.e. geoportals) for their future exploration. In addition, there are resources generated by experts in the field of geographic information that are published on the Web but are not published for their discovery in SDI catalogues or geoportals. Moreover, some works indicate importance of geographic information generated by communities of Web users (i.e. neogeography (Turner, 2006), naïve geography (Egenhofer and Mark, 1995) or Volunteered Geographic Information (VGI) (Goodchild, 2007)) as important source of information for SDI, or at least a complementary source (Craglia et al., 2008; Keßler and Bishr, 2009). These new requirements and the augmentative character of the content produced by the geospatial community require the identification of new approaches for the improvement of search in the Geospatial Web. Web crawling and the Semantic Web have been considered as relevant for this 8Different names are used in the geospatial community when referring to such a system, for example, catalogue services (OpenGIS Consortium), Spatial Data Directory (Australian Spatial data Infrastructure), and Clearinghouse and the Geospatial One–Stop Portal (FGDC). 8CHAPTER 1. CONTEXT AND RESEARCH ISSUES purpose. Today, search engines (SEs) are popularly employed to perform search activity in the Web. In general, a search engine, in response to a user query, returns an ordered list of informative items on potentially relevant Web resources (e.g. Web pages, images) retrieved from a repository. Three main types might be distinguished: directory–based, crawler–based or hybrid (Inthiran et al., 2010). A directory–based SE is based on curated lists (e.g. Yahoo9). The directories existing in the Web usually ensure better precision but Web cover is lower, and frequently domain–oriented. In this sense, geospatial catalogues can be classified as a directory–based SEs. In contrast to a directory–based SE, the repository of a crawler–based SE is created and maintained automatically. Automatic Web crawlers (called also robots or spiders) are responsible for finding, analysing and indexing the Web resource. Considering web resource accessibility through a crawler–based SE, the Web might be divided into three layers: •Surface Web. It is the part of the Web which resources are reachable by web crawlers via hyper-links (Bergman, 2001); •Deep Web. The deep Web is composed of the online databases whose data are exposed via forms, applications or web services (Bergman, 2001); •Invisible Web. The rest of the Web is not easily indexable or does not have valuable content (Price and Sherman, 2001). Domain portals found through a general SE require additional exploration by users to find required resources (Green, 2000). For example, a geoportal is an example of domain portal which requires further interaction to be performed by user in order to access the deep Web resources (Figure 1.5). This is a characteristic of domain–niches, the domains whose resources are of low interest for a general search engine. In other words, the required effort of discovering, indexing and supporting proper search functionality is not compensated by the interest of a typical Web user. Therefore, in terms of specialisation, search engines might be divided into (1) general SEs (for example Google Search Engine10) that offer horizontal search, and (2) domain–oriented SEs (e.g. Google Scholar11) that offer vertical search. The domain–oriented SEs are optimised to support specific characteristics of the domain in terms of thematic search (e.g. in context of medicine (Chakrabarti et al., 1999)), kind of web resource (e.g. web service (Al-Masri and Mahmoud, 2008), picture, video), web cover (i.e. controlled list of web addresses), or combinations of these characteristics. There are works that present a dedicated crawler capable of retrieval and indexing Geospatial Web services (López-Pellicer et al., 2011e; Li et al., 2010). Such crawler can expose deep Web resources of the Geospatial Web (López-Pellicer et al., 2010c). In addition, the research in Web search engines 9http://dir.yahoo.com/ 10http://www.google.com/ 11http://scholar.google.es/ 1.2. MOTIVATION 9 Figure 1.5: Geoportals as access points into multidisciplinary GCI cube for exploring research results. has a long tradition in handling the dynamic aspects of Web content, and the aspects of user search by means of a search engine has been investigated intensively by the Web community (Broder, 2002; Rose and Levinson, 2004). Another research community which technological advances are relevant to the search approaches used in the geospatial community is the Semantic Web community. These technologies are of particular importance for the geospatial community, because the Semantic Web aims to improve the reflection of human reasoning in sharing and processing information contained in the Web resources using automated tools. Such approach requires creation of ontologies – “a formal, explicit specification of a shared conceptualisation” (Gruber, 1993). The semantics that capture the cognitive content of Web resources might be presented in different ways. The easiest way is to add simple metadata, e.g., specially designed tags in XML–based format. The semantics might also be represented as data models via other Web resources that provide conceptual structures, for example as the Resource Description Framework (RDF) (Klyne and Carroll, 2004). The most complex but also the most rich in meaning are ontology–based semantics expressed in the form of RDF+RDFS (RDF Schema) (Brickley and Guha, 2004) or the Web Ontology Language (OWL) (W3C OWL Working Group, 2009). The combination of RDF documents and the Hypertext Transfer Protocol (HTTP) has gained considerable interest in the Semantic Web community, as it allows publishing structured data on the Web as Linked Data (Bizer et al., 2009), and offers logic references to any related resources. The potential of the created Web of Data consists of the identification of a concept via a dereferenceable Universal Resource Identifier (URI) that permits retrieving the description of a concept from Web 10 CHAPTER 1. CONTEXT AND RESEARCH ISSUES as an RDF document. This document may contain references to other documents about the same concept (i.e. identifies its instances) or states the logic relation with other concepts referenced via their URIs. In this simple manner it is possible to create a web of interlaced concepts. Also geographic information has to be accompanied with a knowledge backbone to be appropriately handled due to its peculiarities (Egenhofer, 2002). Therefore, the proper semantic description of geographic information seems to be the first step in the improvement of its usage. A methodology for referencing plain–text annotations to a backbone ontology is being considered by the geospatial information industry (Maué et al., 2009). These additional annotations might be added on three levels, (1) resource metadata (e.g., an OWS Capabilities document), (2) data model (e.g. a Geography Markup Language (GML) Application Schema (Portele, 2012)), and (3) data entities (e.g., a GML file). The formal specifications of concepts from the reference ontologies can be used then for tasks such as semantics–based information retrieval during workflow definition process. The semantic–based solutions have been applied successfully to improve searching within a catalog service, for example by search query expansion to add synonyms or to support multilingual queries (Latre et al., 2009), or event to enable on–demand delivery of geospatial information and knowledge (Yue et al., 2011). 1.3 Problem statement Marvin Minsky (Minsky, 1986) acknowledges that there is no single, perfect principle, and vast diversity enables intelligence. Although he refers to human mind, this statement can be easily brought in other contexts. For example, Spatially Enabled Society requires the applications which use a variety of resources, including spatial data and services offered by SDIs. Also e–science consumes those resources and produces new outcomes that might be issue of incorporation to the existing infrastructures. This aspect of SDI advances some relevant issues. First, an SDI becomes a part of to a broader digital geographic information community, and search task might involve non–SDI resources. Next, the profile of a user of SDI resources evolves as well. There is a general assumption on expert dedication of SDI resources (Boes and Pavlova, 2008). However, the advances in semantic–based solutions dedicated to SDIs (Maué, 2008; Maué and Schade, 2009; Janowicz et al., 2009, 2010) and initiatives that promote an open and massive usage of geospatial data (e.g. Digital Earth (Craglia et al., 2008)), encourage non–expert users to join the SDI user community. Additionally, the Web is used as the technological solution for SDIs. This fact and open standards used by SDIs give possibilities to provide SDIs with an enhanced search for geospatial resources based on the practices of the Web community. Thus, the vast diversity of SDI resources and the need of support for intelligent queries is the basis for the problem statement of this thesis. There is a hypothesis on search improvement within the Geospatial Web in the context of SDI that this thesis addresses. 1.4. RESEARCH QUESTIONS 11 “In order to improve searching for geospatial information and resources in the context of SDI, it is necessary to develop and provide systems which are able to use semantics and content–based heuristics for exploiting the published resources in an automatic manner.” 1.4 Research questions Searching is the process of trying to locate something specific (Beale, 2006). Therefore, considering the perspective of the geographic information community, there might be identified two main categories of searching: •a search for information about a geographic feature, and •a search for geographic information–bearing objects (GIBO) (i.e. data sets that relate to well– defined areas or footprints on Earth’s surface (Goodchild and Zhou, 2003)), which includes search for GIBOs related to a geographic feature. In general, the world can be represented using field–based and object–based views (Goodchild et al., 2007), where a geo–field (i.e. a surface or coverage) represents a set of continuous characteristics of a natural phenomenon, and a geo–object (i.e. entity or feature) represents a set of discrete characteristics of a natural phenomenon. The field/object distinction is closely related to human perception of the world, which is populated with discrete objects (e.g. a named place) and the environment properties are perceived as continuously varying fields (e.g. noise) (Couclelis, 1992). These distinctions in human perception influences the way of searching for information related to a location in the world. Application of the object–based view during search task can be related to the search by place names or textual descriptions of locations (e.g. address). Human perception and SDI characteristics determine searching tasks. In general, there are three basic search patters that might be performed: •Searching for information about a feature. It requires searching through discovered datasets and involves recognition of the searched feature in the collection. •Searching within an SDI for resources that provide information about a feature. It requires searching through an SDI catalogue. •Searching among SDIs for resources that provide information about a feature. It requires searching through a variety of SDI catalogues. In the first scenario, a user has to access a variety of geographic information sources in order to find the required information (see the left site of Figure 1.6). A system that offers enhanced search for information about a geographic feature should be able to access different resources. Considering other search scenarios, the resources should belong to distinct SDIs, and Non–SDI resources should 12 CHAPTER 1. CONTEXT AND RESEARCH ISSUES Figure 1.6: Scenario for searching for information about a feature within various resources, according to a general approach (on the left) and its enhanced version (on the right). Figure 1.7: Scenario for searching within an SDI for resources that provide information about a feature, according to a general approach (on the left) and its enhanced version (on the right). be used as well. Additionally, user requirements should be considered when performing search (see the right site of Figure 1.6). The second scenario corresponds with a typical search task within an SDI (see the left site of Figure 1.7). Here, a user accesses a catalogue to search for resources. Then, the discovered resources are evaluated and selected eventually. The searching results can be improved by applying semantic 1.4. RESEARCH QUESTIONS 13 Figure 1.8: Scenario for searching for resources among different SDIs. and content–based methods to extract additional metadata that permit developing approaches to improve the user experience (see the right site of Figure 1.7). In the last scenario, a user searches for resources across the boundaries of different SDIs. The SDI approach is based on a catalogue which offers an integrated search over several catalogues from different SDIs. In this work, the Web context is considered. New catalogues can be discovered using typical approaches from the Web community, i.e. exploiting search engines to discover at least geoportals for their further exploration (as in the second scenario). In such scenario, the searching can be improved by enabling a system which is able to discover, analyse and index relevant geographic information resources found on the Web. Geoportals are entrance points to SDIs which can be exploited by automatic crawlers (see Figure 1.8). Additionally, non–SDI resources can be considered or these which are not registered within an SDI catalogue traditionally (e.g. a Web site of provider of geographic information resources). A search system based on crawling the Web for the discovery of geographic information resources has some important advantages. First of all, the experience of the Web Service community shows that the publishing-finding-binding pattern applied by SDIs might not be successful, and crawler– based system can handle this issue in some degree. However, this problem can be mitigated by the policy–based character of an SDI, whose members are required to provide (i.e. publish and register) a minimal set of services (e.g. INSPIRE). Additionally, such system can discover and index some relevant geographic information resources which do not belong to any SDI (i.e. to be published in any catalogue), and they might be produced by experts of the geospatial communities or other communities of relevance. Searching within various SDIs in the style of a web search engine seems to be a requirement which is caused by natural evolution in SDIs. Taking into account the incremental character of spatio-temporal data and e-science results (which might be available as raw data), the number of 14 CHAPTER 1. CONTEXT AND RESEARCH ISSUES resources produced by the live SDI community have an augmenting character as well. The user community, even if restricted to specialists (Craglia, 2007; Boes and Pavlova, 2008), is quite broad and not all of them have to be geographic information professionals who “normally know what data are available and where” (Gould, 2007). Advances in offering technological solutions to non–experts from a concrete domain (for example, grids applications, semantic frameworks that are dedicated to non-domain experts), increase number of potential users. 1.5 Methodology The work is divided into separate although complementary applications which investigate possibilities for search improvement in the identified search scenarios in the context of SDI. The applied systematic methodology is related to software engineering. First, the problem is presented and analysed. The solution proposed to the problem is the result of a cyclic incremental development process, which is composed of: 1. Analysis. During the analysis process, the relevant research literature is reviewed to identify advances related to the research issue. 2. Problem specification. It provides a rationale of the motivations or the challenges for the research question. 3. Conceptualisation. A solution is proposed during the conceptualisation stage. 4. Implementation. The conceptualisation guides the implementation developments. 5. Evaluation. The evaluation applies the implementation to a concrete problem and evaluates its usefulness. 1.6 Scope This research work has the following scope: •Feature–based Search. Although Goodchild et al. (2007) identifies two possible views of the world (i.e. the geo–field and geo–object based), the search task is limited only to feature–based approach (e.g. searching for information about ’Madrit’). •Evaluation method. Research literature on evaluation of Information Retrieval (IR) systems identifies four main theories about IR systems (Järvelin, 2011): ranking theory, search theory, information access theory, and information interaction theory. Figure 1.9 presents them as part of the nested evaluation frameworks. In this work, only the two first theories (ranking and search) are considered. The algorithms and methods developed are evaluated using the 1.6. SCOPE 15 Figure 1.9: Nested evaluation frameworks of an IR system (source: Järvelin (2011)). ranking theory. One of the systems developed in this work is evaluated using ranking and search theory. Two other systems are used within real applications. In such case, an empirical study of a prototyped version has not been considered. •Semantic support for search in SDI. One of the relevant research questions is the semantic support in searching within and among catalogues of SDI. For example, the support for thematic search involves vocabulary mappings and/or query expansion. This is not targeted in this work. Here, semantic technologies are used as supporting tools to provide demanded functionality and the conceptualisation process is a secondary activity. These technologies are limited to D2R server and Apache Jena toolkit. •Knowledge representation. In this work, several knowledge representation systems have been used. These systems are mainly simple SKOS (Simple Knowledge Organization System (Isaac and Summers, 2009), a W3C standard for porting knowledge organisation systems to the Semantic Web) vocabularies and RDF/XML graphs. •SDI resources. In general, SDI resources that follow OGC specification have been in the focus of this work. However, there are some exceptions. In the case of the compound geocoding application, services from different SDIs have been integrated, whose interfaces do not necessarily follow the corresponding OGC specifications. A non–SDI resource has been used as well. The system for automatic generation of geographic metadata for Web resources focuses on Web sites of OWS providers. Web sites are also considered by the Geospatial Web Search Engine. •Web metadata. A prototype of the system for automatic generation of geographic metadata for Web resources is limited to HyperText Markup Language (HTML) documents compliant with the HTML 4 W3C Recommendation (Raggett et al., 1999). It uses the header section 16 CHAPTER 1. CONTEXT AND RESEARCH ISSUES to extract the meta elements. Although there exists another manner to embed semantic information within a Web page (e.g. microformats12), the presented content–based methods only process the HTML elements. 1.7 Contributions This work aims to define basic search patterns within the Geospatial Web from the SDI perspective. These patterns are used to show the need for systems that apply semantic and content–based heuristics to enable enhanced search. •First, this thesis identifies three approaches for searching information in the context of SDI. •Second, this thesis develops a compound architecture for searching information about a geographic feature within several datasets. •Third, this thesis describes two approaches that enhance the task of searching for resources within an SDI catalogue. •Fourth, this thesis presents a system for the automatic generation of geographic metadata for Web resources. •Fifth, this thesis presents a Geospatial Web Search Engine tailored to the needs of a non–expert user. 1.8 Thesis structure Due to the interdisciplinary nature of this thesis, each Chapter may include a background section. They are organised as follows. Chapter 2 describes an approach for improving the search for information about a geographic feature. It introduces the concept of geocoding and related terms, such as georeferencing,geotagging and geolocating. Aspects of the quality of a service–based geocoding system are identified (i.e. quality of Web service, quality of spatial data and quality of geocoding process). The application developed for address geocoding is deployed in a real environment. Chapter 3 discusses the improvement of an SDI catalogue by applying semantics and content– based methods. Two different approaches are presented and applied in a real application. The first one shows how semantic Web technologies can be used to improve the precision of spatial search. Linking approaches from the semantic Web community and the Geospatial Web community are discussed. The method proposed is based on the idea of abstraction of a geographic feature from its spatial definition. Modelling the spatial representation of a geo–concept and the idea of geoidentifier 12http://microformats.org/ 2.4. QUALITY OF GEOCODING SERVICE 23 The choice of service is determined by the use case. Free geocoding Web services are appropriate for geotagging (i.e. the act of adding the geographic metadata to any kind of on–line resource) of the local news or incidents (e.g. water supply shortage, planned roadwork) because such information does not require geocoding services of high quality. On the other hand, the systems on which depend public health (Bonner et al., 2003), public security (Ratcliffe, 2004) or environmental services (Ratcliffe, 2001) require high quality of service and data. For example, the speed and efficacy of fire–fighters depend on the information they possess such as the location, accessibility and characteristics of the building in fire (e.g. number of floors, shape, location of entrances and the accessibility to the building, nearby buildings) and fire hydrants. The vast heterogeneity of geocoding services and the specific features of geographic data set up the open problem of provider selection. There are many works in the context of the service discovery and selection. Some proposals need prior service evaluation (e.g. a rating agency (Sriharee, 2006) or a user (Manikrao and Prabhakar, 2005) pre–evaluation), but most of the works in this area use typical QoS features (Yu and jay Lin, 2005; Wang et al., 2006; Tsesmetzis et al., 2008). The research community has also shown interested in services of geographic information (Fallahi et al., 2008; Lan and Huang, 2007) but they include only basic concepts (e.g. coverage) and do not exploit specific characteristics of geographic data in discovery and selection processes (e.g. reasoning based on coverage, quality of geographic objects). 2.4 Quality of geocoding service Quality is defined as “totality of characteristics of a product that bear on its ability to satisfy stated and implied needs” (ISO/TC 211, 2002). In this work, the quality factors of a geocoding Web service are considered. Therefore, the recommendations from the Web Service community and the geospatial community should be considered. Additionally, some specific issues related to the geocoding process will be discussed. 2.4.1 Web service The principal standardising organisations that work on standards related with Web services are the W3C6and the Organization for the Advancement of Structured Information Standards (OASIS, 2012) (OASIS). W3C’s primary activity is “to developing protocols and guidelines that ensure long-term growth for the Web. W3C’s standards define key parts of what makes the World Wide Web works”.7OASIS is “a not–for–profit consortium that drives the development, convergence and adoption of open standards for the global information society” (OASIS, 2012). W3C provide a draft document that describes QoS requirements for Web Services (Lee et al., 2003). However, a recent 6http://www.w3.org/ 7http://www.w3.org/Help/ 24 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY Figure 2.2: Extracting Quality Factors of Web Service (source: (Kim et al., 2011)). and more complete model of QoS for Web services is the Web service quality factor (Kim et al., 2011) (WSQF), a standard approved by the OASIS Web Services Quality Model (WSQM) Technical Committee. This standard defines common criteria to evaluate quality levels for interoperability, security, and manageability of services. A WSQF refers to “a group of items which represent web service’s functional and non–functional properties (or values) to share the concept of web services quality among web service stakeholders. Functional quality reflects how well it complies/conforms to a given design, based on functional requirements or specifications. Non–functional quality refers to how that a service meets non–functional requirements that support the delivery of the functional requirements, such as robustness or interoperability, and the degree to which the service was produced correctly. The WSQFs have been induced from the basic characteristics of Web services (Figure 2.2), and are composed of business value quality, service level measurement quality, interoperability quality, business processing quality, manageability quality and security quality. Depending on business perspective or system perspective, they can be categorised into two groups: (1) the business quality group and (2) the system quality group (Figure 2.3). There is a proposal (Mabrouk et al., 2009) for an extension of the WSQM dedicated to the service environments such as user mobility and context awareness of application services. It provides an overview of several ontologies: •QoS Core ontology (based on an obsolete version of Web Services Quality Description Language (WSQDL) (Lee and Kim, 2006) published by the WSQM Technical Committee in 2007). •Infrastructure QoS ontology. •Service QoS ontology (based on an obsolete version of OASIS WSQM published in 2007). •User QoS ontology. 2.4. QUALITY OF GEOCODING SERVICE 25 Figure 2.3: Structure of Web Services quality factor (source: (Kim et al., 2011)). The service QoS ontology considers dynamic capabilities and domain–specific qualities as well. The proposed model can be a general framework to provide the appropriate ground for several engineering capabilities including QoS requirements engineering and QoS–based service engineering (e.g., service discovery, Service–level agreement (SLA) management, monitoring). Although the model is built on obsolete works of OASIS, the discussed issues remain relevant. Quality of service is also referred in SDI. INSPIRE provide an Implementation Rules document that refers to service quality: the Network Services Performance Guidelines (EC, 2007b). It tries to define minimum performance criteria for the INSPIRE Network Services. Although not all technical specifications of the Network Services are currently available, these guidelines are intended to be applicable to all and to be compatible with the Network Services Architecture Document (INSPIRE NS DT, 2008). These general guidelines have been defined by revising existing standards and recommendations (including the described above), and identify the following attributes of QoS for the INSPIRE Network Services: Performance, Reliability, Capacity, Availability, Security, Regulatory, and Interoperability. The concepts definitions correspond with those provided in Lee et al. (2003). 2.4.2 Spatial data quality van Oort (2005) identifies several reasons for concerns about spatial data quality issues, as follows: •There is an increasing availability, exchange and use of spatial data. •There is a growing group of users less aware of spatial data quality. 26 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY •GIS enable the use of spatial data in all sorts of applications, regardless of the appropriateness with regard to data quality. •Current GIS offer hardly any tools for handling spatial quality. •There is an increasing distance between those who use the spatial data (the end users) and those who are best informed about the quality of the spatial data (the producers). There are intensive standardisation works on handling quality within the geospatial domain. The ISO standards, for example, provide quality principles and define specific concepts (ISO 19113 (ISO/TC 211, 2002)), define principles for quality evaluation (ISO 19114 (ISO/TC 211, 2003)), and provide description of quality assessment methodologies (ISO 19138 (ISO/TC 211, 2006)). As for data quality, ISO 19157 standard revises ISO 19113, ISO 19114 and ISO 19138, and defines a set of measures for the spatial data quality elements identified in ISO 19113. At the time of writing this thesis the standard is still under development (i.e. at enquiry stage8). Sharing and reusing spatial data require paying special attention to quality of spatial data, therefore this issue is relevant in any SDI. Quality has to be considered from different perspectives in an SDI. There might be different viewpoints used to describe quality (Garvin, 1988; Jakobsson, 2006). In his thesis (Jakobsson, 2006), Jakobsson argues that quality management viewpoints are important from the SDI’s point of view. He discusses geographic quality concepts using these four viewpoints (Figure 2.4): •Production–centred viewpoint. This perspective focuses on the variations in the production process where the most common measure is the number of defective or non–conforming products. •Planning–centred viewpoint. This perspective focuses on the characteristics of products. •Customer–centred viewpoint. This perspective focuses on the value of products and services to the customer. •System–centred viewpoint. This perspective takes into account all stakeholders who are influenced by the organisation or its products oriented quality. In the case of INSPIRE, Article 17 of the INSPIRE Directive (EC, 2007a) says: “Each Member State shall adopt measures for the sharing of spatial data sets between its public authorities ... for the purposes of public tasks that may have an impact on the environment.” Therefore, an effort within INSPIRE is dedicated to the development of some methods for assessing, measuring, reporting and controlling spatial data quality. These aspects have been considered by the European Spatial Data Infrastructure with a Best Practice Network (ESDIN) project9supported by eContent+ programme. 8http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=32575 9http://www.esdin.eu/ 2.4. QUALITY OF GEOCODING SERVICE 27 Figure 2.4: Different approaches to geographic information quality from the quality management viewpoint (source: (Jakobsson, 2006)). Data quality is also considered by the FGDC CSDGM standard (which incorporates the Spatial Data Transfer Standard (SDTS) (FGDC, 1998b) that contains a section on spatial data quality elements) and the NAP standard. There is a remarkable agreement among the documents on the elements of spatial data quality. Each of the standards that approach that question describes the same core elements: •Attribute (Thematic) Accuracy. CSDGM and SDTS use term “Attribute Accuracy”, and ISO 19115 refer to the same content as “Thematic Accuracy”. It can be defined as “an assessment of the accuracy of the identification of entities and assignment of attribute values in the data set.” (FGDC, 1998a). •Completeness. It refers to “information about omissions, selection criteria, generalization, definitions used, and other rules used to derive the data set” (FGDC, 1998a). •Lineage. It refers to the “information about the events, parameters, and source data which constructed the data set, and information about the responsible parties” (FGDC, 1998a). •Logical Consistency. It refers to “an explanation of the fidelity of relationships in the data set and tests used.” (FGDC, 1998a). •Positional Accuracy. It is “an assessment of the accuracy of the positions of spatial objects” (FGDC, 1998a). 28 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY •Temporal Accuracy. It is usually defined as “accuracy of the temporal attributes and temporal relationships of features” (ISO/TC 211, 2002) 2.4.3 Geocoding quality Geocoding quality has been investigated intensively in the fields related with spatial analysis, and there is a lot of work on the evaluation of geocoding system accuracy. In general, the typical metrics used to determine fitness-for-use are as follows (Goldberg et al., 2010): •Match–rate. It can be compared to the precision and recall of IR systems, and it refers to the number of input addresses that a geocoding system was able to match. •Match type. It is the level of geographic feature matched to e.g., parcel centroid, postal code, street address. •Match certainty. This value describes the level of similarity and/or likelihood of a match between the input address and the address associated with the matched feature input derived either probabilistically or deterministically. •Spatial accuracy. It is defined usually as an average values of distance and direction “from truth” by comparing computed output locations and known locations for a subset of the data. However, these metrics do not properly represent the uncertainty value of geocoding results in order to be adequately used in spatial analysis (Zandbergen, 2009). A typical geocoding process is composed of (1) data cleaning, (2) feature matching and (3) feature interpolation. Figure 2.5 outlines a generalised workflow of this process. As geocoding process has influence on geocoding quality, each component of the geocoding workflow has influence on geocoding quality: •The quality of the input data which is the text describing a location. Figure 2.5: The Generalised abstraction of the geocoding process (source: (Goldberg, 2008)). 2.5. COMPOUND GEOCODER 29 Figure 2.6: The alternative paths in geocode production (source: (Goldberg et al., 2010)). •The input data parsing and normalisation algorithms which identify the pieces of the input text and transform them to standard values (Davis and Fonseca, 2007). •The feature matching algorithms which identify candidate matching geographic features in the reference data sources. •The feature interpolation algorithms. Some works attempt to quantify an overall propagated uncertainty value of geocoding results (Davis and Fonseca, 2007; Zandbergen, 2009; Goldberg et al., 2010). An interesting approach to geocoding accuracy is presented in Goldberg et al. (2010). A geocoding process is seen as “a decision tree with multiple potential outcomes at each level (transformation) that guide the set of choices available to all subsequent levels” (see Figure 2.6). The authors provide discussion on quantitative accuracy metrics to describe the quality of geocoded data from the perspective of the spatial certainty, and propose to report the results of the geocoding process as spatial probability distributions. This method assumes availability of several quantitative factors to describe the spatial–temporal aspects of accuracy and uncertainty for each component. However, the authors admit that the quantities are not well defined, which is stated as future work. 2.5 Compound geocoder Geolocating process demands spatial information of a wide range of types. Therefore, a compound approach seems to be a suitable architecture model to provide an enhanced geocoding functionality. The proposed compound geocoding architecture allows the building of hybrid solutions composed of different services (e.g. geocoding, gazetteer or cadastral service) enhanced with a geocoding layer if necessary. The task of geocoding service selection exploits geographic ontologies. The Administrative Unit Ontology (AUO) (López-Pellicer et al., 2008) is used for (i) provider selection and (ii) data integration. This approach may increase the flexibility and adaptability of applications. In the case 30 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY Figure 2.7: The selected characteristics of geocoding service. of the public services, it provides access to different services, national and local, in a transparent manner and ensures the use of updated data. First, the characteristics of geospatial services considered in this work for service selection are presented. Then, a patter–based method for service integration is briefly outlined. Finally, the architecture proposed is outlined. Geocoding service quality The proposed architecture allows the service selection according to the use case requirements. Therefore, the proper description of each source is vital for the behaviour of the whole system. Figure 2.7 presents the features to consider during the evaluation of providers. The obtained values are stored in RDF files with corresponding semantic annotations. These values are essential for the service selection and the decision making process. The main features of each geocoding service are (1) the coverage, (2) the type of content and (3) the type of spatial object. The first two features are always given by the provider or might be indicated by the name of the service. The first one defines the area in which offered data are situated. Usually, this area corresponds with the unit or set of units from political division of territory, because the majority of georeferencing Web services are provided by public administrations from local and central level. The solution proposed is based on the use of AUO as the ontology of political organisation of the territory. Using instances of classes which are specialisations of jurisdictional geographic object concept from AUO as value of the coverage allows reasoning on the semantic relations among them, rather then relying on their spatial objects. For example, the province of Zaragoza has a set of municipality members and is part of Spain. All municipalities which belong to Zaragoza province are also members of the Aragon Autonomous Community. If searching for services which provide data from the province of Zaragoza, system will also consider those services whose coverage equals the coverage of to Aragon, Spain or the world, or even might return a set of services whose 2.5. COMPOUND GEOCODER 31 coverage combination corresponds with the required coverage, e.g. local services of all municipalities of province of Zaragoza. The type of content strictly depends on the types of geographic feature provided by service. The last one, the type of spatial object, indicates the list of provided types of spatial object, such as point, polygon or 3D entity. For example, the Cadastre Service of Spain10 (Servicio de Catastro de España), as the name of the Web service indicates, has the coverage of Spain and offers centroids of parcels. Google Maps, according to the online documentation, has world coverage and its type of content is street address geocoded via point. From the analysis of spatial data, it is possible to obtain two additional indicators (of range 0 – 1): (5) the reliability and (6) the precision. The reliability indicates the capacity of representation of elements of physical world by the content. The service that offers all elements of the real world has a reliability value equal to ’1’ (100%). The indicator of precision informs about the average positional error of the whole dataset. It is important to note that this indicator may be influenced by the difference between the provided spatial object and the search one. For example, using cadastral data for the address geocoding, there will be a decrease in spatial data precision. The above presented factors can be used to express the quality of a spatial dataset as a result of comparison with a baseline dataset. Such relative estimation requires semantic accuracy of compared objects. This aspect is related with (4) the result accuracy, a typical geocoding quality factor (i.e. the match type). It should not be misinterpreted as “data accuracy”, the term commonly used in literature to describe positional accuracy. Result accuracy is estimated for each source via analysis of the source data model and the application domain model. It indicates the capacity of source to fulfil the domain model, and is represented via the last domain model field which the source might score. Frequently, values of the reliability and/ or the precision of spatial content can vary in function of the area of relevance (e.g. new suburbs might be even omitted). Such feature, called the granularity, might be obtained from the exhaustive evaluation of spatial content along with its semantic analysis (e.g. by distinguishing types of geographic features as: cities of population more than 500.000, cities of population more than 100.000, towns of population more than 10.000, villages, and hamlets). Mediator layer Services selected to be used as the external resources for information search may vary in terms of technological solutions used by providers. Adding a new, common functionality to a service or over a set of existing services, requires creation of a mediator service. Depending on user needs and characteristics of the service which should be adapted, one of the following design patterns might be used to develop a mediator service: adapter or façade. The adapter pattern allows the user to access to functionality of an object via known interfaces and/or adopting message channel (Gamma et al., 10http://ovc.catastro.meh.es/ 32 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY Figure 2.8: Workflow for the metadata generation of a mediator service. 1994; Hohpe and Woolf, 2003). In the majority of cases, the mediator service will implement this pattern to encapsulate the invocation of the original service according to specific requirements. In addition, it may hide the logic of multiple requests or error handling. The façade pattern provides “a unified interface to a set of interfaces in a subsystem. Façade defines a higher–level interface that makes the subsystem easier to use.” (Gamma et al., 1994). In terms of services, this approach addresses subsystems which combine several services and allows their logic to be concealed. For generalisation purposes, an abstract method for service integration into an OGC mediator layer is proposed. The method applies the adapter or façade design patterns to create a mediator service. The implementation of a mediator service requires the identification of the source service (or a set of services) (i.e. the Adaptee Service) and the functionality which should be provided by the target service. The target service is an OGC service which may offer the desired functionality. The method produces several metadata documents which will be used as guidance during implementation of the mediator, i.e. the Capabilities document that conforms to the selected specification and some additional service metadata (e.g. the ToS of the Adaptee Service). The first step of the proposed method is the selection of a base service (or services) to be adapted (see Figure 2.8). A simple metadata template should be filled with information found in the available documentation of the chosen service. Although this might sound rather straightforward, in practice it is not so simple. Popular Web services (e.g. the Google service family) lack documentation in a standardised format, for example a Web Service Definition Language (WSDL) document. Commonly, they only provide human–readable documents, an API, or sample code that the service provider publishes for developers to use. Less frequently a WSDL description is provided. If there is not a WSDL file, this method requires the creation of one during the documentation analysis. The created Adaptee WSDL File and the fulfilled service metadata template that produces the Adaptee Service Metadata are the result of the documentation analysis. 2.6. ADDRESS GEOCODING FOR URBAN MANAGEMENT 39 Listing 2.1: Example of RDF description of a mediator service which uses a set of the Cadastre Services of Spain. 1@prefix st: < http: // idee .unizar . es /SW/ services / geoservicesotopology # >. 2@prefix sc: < http: // idee .unizar . es /SW/ services / geoservice / characteristics # >. 3@prefix gsa: < http: // idee . unizar .es/ SW / services / geoservice / addressextent #>. 4@prefix scat: < http: // idee .unizar . es /SW/ services / geoservice / servicecat #>. 5@prefix gsc: < http: // idee . unizar .es/ SW / services / geoservice / category #>. 6@prefix ag: < http: // idee . unizar . es / SW / Onts / AU/ AUSpain . owl # >. 7@prefix rdf: < http: // www .w3. org /1999/02/22 - rdf - syntax - ns #>. 8 9scat:catastrospain2008 rdf:type gsc:AddressGeocodingService ; 10 rdfs:label " Servicio de Catastro "@es ; 11 sc:reliability "1"; 12 sc:precision " 0.8"; 13 sc:resultAccuracy gsa:portal ; 14 sc:spatialObject st:point; 15 sc:coverage ag:ESP_STATE. Parameter/Web service Coverage Reliability Precision ResultAccuracy IDEZarSG municipality 0.98 1 Portal Service of Zaragoza (baseline dataset) GoogleMaps Service World 0.96 0.99 Portal Service Cadastral Spain 1 0.85 Portal Service of Spain (baseline source) CartoCiudad Spain 0.90 0.98 Portal services Table 2.1: Geocoding parameters of used Web services. 2.6.3 Geocoding framework The developed service has been applied as a generic component in the urban management systems of Zaragoza city council. Figure 2.13 presents an overview of the applied domain data model in this framework. The instances of this service are dedicated to diverse applications whose requirements are stated by the profile descriptors. Figure 2.14 shows an overall view of the compound architecture of geocoding service and some examples of types of client applications that use the service. The unified access to georeferencing Web services (e.g. the Cadastre Service) is offered by connectors that implement the AdvancedGeocoder interface and hide the request workflow in the case of the Cadastre and the CartoCiudad services. This interface is implemented also by the Geocode Wrapper which accesses INE local database, the information about census area units in Spain. The data are stored locally due to the fact that it is not provided through any Web service and the data must be downloaded as CSV files from National 40 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY Figure 2.13: Application of the domain data model. Figure 2.14: Layer view of the compound geocoding service and client components. Statistics Institute Web page17 (Instituto Nacional de Estadística, INE). As the INE information is official and complete it is therefore used as a reference point in case of result data ambiguity. The cache component improves the performance of the system and the logger (Log) permits tracking of its behaviour and offers feedback information on user request profile which is a valuable hint in the cache strategy. The simplified BasicGeocoder interface is offered via HTTP protocol and it is the request point for the majority of client applications. 17http://www.ine.es/ 2.6. ADDRESS GEOCODING FOR URBAN MANAGEMENT 41 Figure 2.15: Introduction and publication of urban incidents. 2.6.4 Application An example of these applications is the system to support management of local incidents in Zaragoza locality used by the city council. This system is used by the council workers to introduce the local incident information, which is provided to citizens via an official online portal. Figure 2.15 shows the general overview of the application architecture along with the screenshots of its Graphical User Interface (GUI) components. The form of the Input Component allows the user to introduce a new incident and the results of address geocoding are presented as an ordered list ordered via recommendation based on street name and type matching. The selected, normalised street address, its spatial coordinates (a point) and information about the incident (time interval, incident type, etc.) are saved in the local incident database, that is then used by the GeoRSS service (Reed, 2006), which publish new information on inscription lists. This information might be also visualised on the local incident map of the official Web page of Zaragoza city council. 42 CHAPTER 2. ENHANCED SEARCH FOR A GEOSPATIAL ENTITY 2.7 Summary This Chapter has proposed an approach to enhanced search for a geospatial entity from the perspective of traditional geocoding. The compound geocoding proposed ensures the improvement of geocoding results (e.g. reliability, spatial data quality) thanks to the use of different geographic information suppliers. The use of multiple sources involves the development of ontology for describing the geocoding sources. Moreover, the use of ontologies yields an advanced architecture in terms of extensibility, flexibility and adaptability. The framework for geocoding service selection permit the development of a methodology to geocode diverse categories of data types (e.g. spatial features, points of interest), which is an essential functionality of a geolocating service. In this context, there is a strong demand for a generic search model that involves the formalisation of search model. Additionally, the knowledge integration system adds new issues to the gazetteer concept. Usually, the spatial data of gazetteers is obtained from geocoding processes and comes from diverse data sources. Nevertheless, data models of gazetteers do not offer information about spatial data accuracy and its origins. The principal disadvantage of this approach is the need of implementation of a mediator for each source. Although the implementation permits adaptation to changes (e.g via pluggable connectors), the effort of mediator creation is significant. Chapter 3 Content–based generation of the semantic characterisation of geospatial resources 3.1 Introduction This Chapter presents methods for semantic characterisation of geospatial resources that go beyond the specifications from geospatial community. It shows applications where the search result can be improved by more precise resource descriptions. The descriptions are generated automatically with consideration of the content as a geospatial resource. As a variety of possible approaches might be regarded within the geospatial community, two examples of such applications are investigated. The first (Section 3.2) focuses on providing more precise spatial objects when interacting with end users. Here, semantic technologies are applied to improve the precision of the spatial search within a services catalog. The second application (Section 3.3) is dedicated to imagery layer identification in WMS services. The method developed has been implemented as a WPS service. The prototypes developed are applied in applications where usability is affected (Section 3.4). Finally, some conclusions are presented in Section 3.5. 3.2 Geoidentifier One of the main advantages of applying linking–based approaches for referencing geographic features is the maintenance of the abstraction from their spatial representations. Usually, geographic features are characterised by the blurriness of their footprints. Topological elements of the physical world usually lack well defined conceptual boundaries and, consequently, this influences their spatial 43 44 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES Figure 3.1: Modelling the spatial representation of a geo–concept. Spatial A geo–concept from top–level ontology represents a general understanding Reference of a concept. The definition of its spatial realisation is quite difficult and comes as a fuzzy definition (e.g. a spatial interpretation of a river) Spatial Definition of spatial realisation of a geo–concept linked to Interpretation a concrete domain. Footprint The definition of a perfect footprint of a geo–concept entity (linked to a concrete domain). Its computational representation might not exist. Spatial A computational representation of the Footprint. Object It is not a unique identifier of a geo–concept entity. It permits the visualisation of it and reasoning on spatial relations. Table 3.1: Modelling the spatial object. definition (e.g. rivers, chains of mountains). What is more, the computational representation of a geographic feature footprint is limited due to the resolution limits. Any geographic feature may be represented via different spatial objects which depend on the system application, the data model, and the applied technological solution. For example, a point is the best choice as a location reference while a polygon is for landscape visualisation. Therefore the spatial object should be interpreted as one of the possible representations of an entity. Figure 3.1 presents the global view of spatial reference definition, and Table 3.1 provides an overview of the process of conceptualisation. The objective of this work is to present how an SDI might take advantage of best practices from the Semantic Web. An administrative geography (AG), i.e. an ontology of political organisation of the territory, published as Linked Data will be introduced as a relevant element of an SDI, since it permits (1) reasoning on logic relations among administrative units, and (2) accessing different representations (including footprints) of a geographic feature. Such ontology can improve functionality of a services catalogue deployed in an SDI by applying geographic reasoning. One of the principal characteristics of an OWS service is its geographic extent provided by the publisher as part of its 3.2. GEOIDENTIFIER 45 descriptive metadata (i.e. getCapabilities response). According to the INSPIRE Metadata Implementing Rules (INSPIRE DTM and EC/JRC, 2010), the geographic location is defined as minimum bounding box (MBBOX). The descriptive metadata are used by the discovery service to answer the user requests, and a spatial restriction is one of the requestable parameters. Since services from SDI are provided by public administrations, frequently, their geographic extent corresponds with the administrative area of the provider. The usage of MBBOX introduces false positive in the catalog response because the administrative areas are not rectangular. A catalog provided with knowledge about the hierarchy of administrative units and accurate spatial objects of each administrative area might increase considerably the precision and recall of the requests, which is especially important for applications based on on–the–fly data integration. The rest of the Chapter is organised as follows. First, the state of art in linking geographic features in the Semantic Web community (starting from Linked Data approaches to deal with geographic information, and ending with examples of semantic geographic platforms existing on the current Web) and the geospatial community. Next, the infrastructure for the improvement of services catalog is presented. 3.2.1 Linking geographic features Geographic features are managed differently in the Semantic Web and in the Geospatial Web. The first community treats geographic features as the additional contextual information, which might link to some other concepts. For the Geospatial Web, the concept of geographic feature is the core element of a geographic platform. However, the Geospatial Web has focused on the interoperability issues maintaining the boundaries among the concepts from different geographic sources. The Semantic Web As the contents of SDIs are concealed for common Web users, the Web community has created proper solutions applying successful linking approaches, such as geographic Web platforms (e.g. semantic geocoding service of GeoNames (Wick, 2012)) and for publishing geo–data (e.g. LinkedGeoData (Auer et al., 2009b)). Also, the Linked Data has been applied successfully in spatial solutions (e.g. DBpedia Mobile (Becker and Bizer, 2008, 2009)). Both approaches, standardised OGC services and Linked Data, might complement each other. For example, a created ontology of geographic features might link instances of the same geographic feature across different sources. This framework would provide an integrated view of geographic features rich in logic and spatial information. The richness of geographic feature description, direct or provided by linked sources, might be helpful in dealing with conflation for database integration, the well known problem from the database field (Dolbear and Hart, 2007). Additionally, such unified ontology might be used for defining and publishing complementary logic relations among geographic features (e. g., a representation of different territorial organisation) instead of creating new instances of OGC services. As 46 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES for the Semantic Web community, the publication of geographic ontologies by official providers (i.e. a public administration organ) according to Linked Data principles might be a valuable source of references, for example the Administrative Geography of Great Britain (Goodwin et al., 2008). Nowadays, it is possible to generate Linked Data from a variety of data sources, and support a wide variety of data representation and serialisation formats. There are two main approaches for generation of RDF linked data from existing Web content and deploy them on the Web. The first one maps the data gathered in relational data bases to one or more ontologies / schemas producing RDF Linked Data, and the other approach uses RDFizers, i.e. tools capable of extracting data from one or more sources, which then are mapped. D2R Server (Bizer and Seaborne, 2004) is one of the first tools for publishing the content of database. OpenLink Virtuoso (Erling and Mikhailov, 2009) is more complete technological solution and it supports both approaches. The most popular way of accessing to Linked Data is via a SPARQL endpoint or a REST Web service (Alarcón and Wilde, 2010). Linked Data clients vary from simple RDF Browser to a graph visualisation. Also, there are techniques for requesting RDF data from different endpoints providing a transparent on–the–fly view to the end user (Langegger et al., 2008). The most important initiative related to creating and publishing interlinked contents on the Web is the W3C Linking Open Data (LOD) project (W3C SWEO, 2012; Heath and Bizer, 2011) founded in January 2007. In May 2007, the number of LD datasets in the LOD cloud diagram was 12 and in September 2011 it rose to 295 (Cyganiak and Jentzsch, 2011). Since on–line contents involve geographic information, geographic features have become part of Linked Data datasets, such as DBPedia, where geographic information has been extracted from Wikipedia (Auer et al., 2008; Becker and Bizer, 2009). In May 2007 there were only six geographic datasets in the LOD cloud diagram, in July 2009 eight, and in September 2011 the number of geographic datasets rose to 31 providing more than 6 billion RDF triples (Cyganiak and Jentzsch, 2011). For example, the LinkedGeoData (Auer et al., 2009b; Stadler et al., 2011) and GeoNames (Wick, 2012) datasets aim at adding geo–semantic meaning to the Web. LinkedGeoData offers a Linked Geo Data Knowledge Base with RDF descriptions of more than 350 million spatial features from the OpenStreetMap database. GeoNames provides a set of REST Web services to access geographic features (about 8 million unique features). From the point of view of the LOD community, a core of interlinked data about geographical locations is in DBPedia since both geographic datasets are linked to it (Heath and Bizer, 2011). However, it is because of the historical reason (i.e. DBPedia is one of the first LD datasets) and the applied linking paradigm (i.e. unidirectional view). An example of applying Linked Data principles in location based solution is the DBpedia Mobile (Becker and Bizer, 2009), a location–aware client for the Semantic Web for mobile devices. The current user location is used to extract corresponding datasets from the underneath DBpedia database which are interlinked with various other location–related datasets. 3.2. GEOIDENTIFIER 47 An interesting proposal is Triplify (Auer et al., 2009a) which supports circle–based spatial requests. This system directly uses the DB Views model as a base for creating the RDF documents and URLs of published datasets, which facilitate the development. The underlying data base is responsible for processing spatial and semantic queries which are encoded explicitly in the requested URL. The spatial query permits users to retrieve the geographic features located in a circular region defined via a point and radius added to the requested URL. This system enables limited spatial query notwithstanding the fact that the Semantic Web techniques can not take advantage of this facility. The idea of linking geographic features to create the Web of Data influenced the development of geographic Web platforms, such as Yahoo! GeoPlanet (YDN, 2012) or GeoNames (Wick, 2012). Both of them belong to a new branch of geocoders, the semantic geocoders, which return URIs to identify uniquely the named places instead of standardised textual description and location reference (e.g. a point). Although, they use the idea of linking, they are not following the pure Linked Data approach. GeoPlanet uses URIs to identify the named place which permits users to retrieve its semantic description, however, the platform uses a simple XML file instead of RDF. Additionally, the important spatial relations (e.g. child,neighbour,siblings) are encapsulated into the URI definitions. GeoNames is almost Linked Data based. It also uses unique identifiers of concepts to identify the named places. The RDF description of features contains spatial relations defined in the published OWL reference ontology1. However, GeoNames distinguishes the concept from the descriptive document. The feature (i.e. concept) is identified via a URI but the geonames server uses 303 redirection to display its location on a map. The RDF description is available by adding the /about.rdf at the end of the feature URI2. The spatial requests supported by the presented solutions of the current Semantic Geo Web are based on a branch of predefined spatio–logic relations (e.g. near–by,belongs–to,child,siblings). Since it is impossible to express all spatial relations among geographic features via definition of logic relations, the Semantic Web needs to use a spatial representation of features. Currently, the spatial objects usually used in the Semantic Web are limited to points or MBBOX eventually. The complex spatial requests that mix spatial objects and spatial relations defined in a rich ontology still remain the open issue. The Geospatial Web The requirement of unique identifiers for geographic features, geoidentifiers, in the geospatial community has been presented from its beginning. Any Geospatial Web framework which publishes information about geographic features uses unique identifiers (within this framework at least), and it might be seen as a source of geoidentifiers. Therefore, any gazetteer (Hill, 2006) or OGC Web Feature Service from an SDI might be such a source of geoidentifiers. Currently, there are several 1http://www.geonames.org/ontology/ontology_v3.01.rdf 2For example, the town ’Embrun’ in France has associated these two URIs:http://sws.geonames.org/3020251/ and http://sws.geonames.org/3020251/about.rdf 48 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES instances of WFS services in the SDI of Spain, which frequently contain instances of the same geographic feature. The services model the geographic feature in different manners and use different identifiers that usually are derived from keys of relational databases; therefore, there are problems of entity identification among different contents, and as a consequence, common problems of data integration. One of the earlier proposals from the OGC community to apply common geographic identifiers for linking purposes is a framework based on Geolinked Data Access Service (Schut, 2004a) and Geolinking Service (GLS) (Schut, 2004b). This approach is dedicated to publishing geographically linked information (e.g. statistical data) separately from its spatial representation (spatial objects). Since this proposal is based on dataset merging from different sources by using a linkage field defined in the sources, it fixes geographic data to only one source of spatial representations and the geoidentifiers are used only as syntactic links. In practice, it can be seen as a technological facilitation for data publishers. The proposal evolved into the OGC Geographic Linkage Service (GSL) (Schut, 2009) (the result of the Geolinking Interoperability Experiment3), and finally into the OGC Georeferenced Table Joining Service (TJS) standard (Schut, 2010). A proposal for providing an integrated view across distributed services is the EuroGeoNames project implemented as an INSPIRE compliant service (Jakobsson and Zaccheddu, 2009). Apart from defining the data model to be followed by all community members, it provides some rules for identifier definition. The named place identifier has to be composed of (1) its name, (2) the two–letter ISO 3166 code (ISO, 2007a) and (3) a code generated according to the Base36 encoding system (Oscarsson, 2001) (e.g. “2YC67000B”). However, these identifiers still remain unique only in this distributed gazetteer. Interlinking the corresponding instances of the same geo–concept entity across different providers might be interesting for the Geospatial Web. It might be the way of adding logic relations among geo–concept entities and avoid the necessity of providing a new separate platform. For additionally, it can be useful for the improvement of search in an SDI. 3.2.2 Spatial search Using linking principles for search in the geospatial domain has some advantages. One of the advantages is the accessibility to multiple representations or models of a geo–concept entity. It includes access to other spatial objects. Traditionally, when searching with explicit spatial restrictions (i.e. using spatial objects), MBBOXES are used. Some applications support circle–based spatial queries (Auer et al., 2009b; YDN, 2012). Nevertheless, the circle defined might be seen as an approximation of a MBBOX. During the retrieval process, spatial indexes created on georeferenced elements are exploited. The most popular indexes used in spatial data bases apply boxes due to performance reasons (Manolopoulos et al., 3http://www.opengeospatial.org/projects/initiatives/geolinkie 3.3. SEMANTICS OF THE WMS LAYER 55 to make data accessible and convey them in an easier–to–perceive way (Iosifescu-Enescu et al., 2010). Especially when presenting information to a non–expert audience, visualisation of the data improves the understanding of the situation at hand. Other applications of remote sensing products cover change analysis for monitoring and tracking the type and rate of landscape changes (Julea et al., 2010), urban environment modelling (Krauss et al., 2007), or assessing geospatial information quality (Skirvin et al., 2004). With the constant improvement of technologies in high–resolution satellite remote sensors, GPS systems, databases and geoprocessing sources, there are nowadays increasing amounts of imagery and gridded data. Additionally, thanks to the development and increasing importance of SDI, the availability and accessibility of these data through standardised and interoperable Web services have increased exponentially in the last years. The OGC WMS and WCS service specifications, provide the means for the implementation of services which offer visualisation and download of imagery data in well–known formats such as HDF–EOS (Larry Klein, 2007), GeoTIFF (Ritter and Ruth, 2000), DTED (NGA, 1996), NITF (DoD, 2006) or GML. However, it can be observed that there are two main problems that become obstacles for the access to imagery data on the Web. On the one hand, SDI catalogues (the services provided by SDIs to locate data and services) deployed at national, regional or local levels do not necessarily register all the services providing access to imagery data on the Web. On the other hand, it is not easy to automatically identify whether the data offered by a Web service is directly imagery data or not. With respect to the first problem, the discovery of Web services not directly subscribed in SDI catalogues, some researchers have proposed different strategies based on crawling the Web (Li et al., 2010; López-Pellicer et al., 2011e). However, the second problem, the automatic categorisation of the content offered by services has received little Listing 3.2: SPARQL request pattern for service selection via an MBBOX. 1PREFIX svont: < http: // purl . org / iaaa / sw/ gsw / ont /app / gservont . owl #> 2PREFIX agont: < http: // purl . org / iaaa /sw /gsw / ont / app / agont . owl #> 3PREFIX rdf: < http: // www .w3. org /1999/02/22 - rdf - syntax -ns #> 4PREFIX dc: < http: // purl .org /dc/ elements /1.1/ > 5PREFIX sf: < http: // purl . org / iaaa / sw / gsw / app / gfun > 6SELECT ? s WHERE { ?s rdf:type svont:service ; 7dc:coverage ?x. 8?x rdf:type agont:au; 9agont:bond -1000 ?g. 10 FILTER ( sf:INTERACT (?g , $ BBOX )) } UNION 11 SELECT ?s 12 WHERE { 13 ?s rdf:type svont:service ; 14 dc:coverage agont:Non ; 15 svont:bbox ?g. 16 FILTER ( sf:INTERACT (?g , $ BBOX )) 17 } 56 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES attention until now. The purpose of this work is to investigate this second problem and propose a method for the automatic analysis of WMS services in order to detect if they contain imagery data, i. e. aerial photos or orthorectified satellite images of high resolution. WMS services are considered because their availability is higher on the Web. Currently, the number of WCS service instances is very low in comparison with WMS services (López-Pellicer et al., 2011a). Additionally, as WCS services provide full access to data, complex issues about access rights usually restrict their public deployment. Various heuristic methods for the automatic analysis of WMS services have been investigated in this work. First, a description–based method has been developed. It exploits information offered by the service provider. Then, a content–based method has been investigated. The characteristics of the information that should be detected (i. e. orthoimages) derive from the requirements on radiometric and spatial resolution of sensors that produce it. Therefore, the effort focuses on deterministic methods that exploit image features generated by these kinds of sensors. Additionally, an effective algorithm for the gathering of spatial information from OGC WMS services is an important issue of this content–based method. In the context of this work, an effective retrieval algorithm for content– based analysis means an algorithm for collecting a representative set of fragments of an image (i.e. not empty image fragments), which enables layer analysis. The procedure developed has been published as a geoprocessing service, which accepts the URL of a WMS Capabilities document and returns a list of those of its layers which offer orthoimages and the scale at which the images have been discovered. This geoprocessing service has been applied within the Virtual Spain project6to develop a catalogue of orthoimages and associated applications. The rest of this section is structured as follows. First, the existing related work is outlined. Next the method proposed for the analysis of WMS services is presented. Then some experiments are performed over a crawled collection of WMS services and the efficiency of this method is discussed. Finally, the publication of the proposed method as a Web service compliant with the OGC WPS specification is presented. 3.3.1 Related work The accessibility of geospatial resources influences any geospatial–based tasks related to geospatial processing. For example, on–demand data production (Mansourian et al., 2008) assumes discoverability of proper geospatial resources as an input. The discovery of resources within an SDI is based on the DL paradigm (Béjar et al., 2009). For instance, Li et al. (2011) present a framework for searching over multiple standardised catalogues, which can provide an integrated search across local and regional SDIs. However, not all Web resources valuable for these kinds of geospatial processes are part of an SDI, e.g. OGC Web services offered by a public administration but not published within any SDI, or resources produced by Web users. In these new scenarios, the automatic discovery 6España Virtual project (Virtual Spain project): http://www.españavirtual.org/ 3.3. SEMANTICS OF THE WMS LAYER 57 of geospatial resources on the Web has recently gained interest within the geospatial community (Li et al., 2010; López-Pellicer et al., 2011e). This new approach might provide users with a wider range of geospatial resources. Although standardised geospatial services are thought to be self–descriptive, some researchers indicate that there is a lack of good practices. For example, it is common to find inconsistencies between metadata in registries and the metadata offered directly by the service (Wu et al., 2010). As automatic discovery on the Web relies mainly on resource analysis, it might motivate providers to ensure more accurate descriptions. For example, in the case of OGC services, it might be supposed that if the discoverability of a resource depends on the quality of its Capabilities document, the providers would put more effort in the future to generate more valuable information in comparison with the current practices. In the context of this work, WMS instances crawled on the Web are used as input in the experiments to test the performance of the method proposed for filtering orthoimages. As these services have been found on the Web and are deployed by different providers, they represent a realistic scenario that combines heterogeneous metadata of different quality levels. Although resources are usually described in conformance with a specific description model, this does not necessarily imply that all potentially interesting information to the user is provided. For example, the WMS Capabilities document does not contain a field to state the data presentation scale, i. e. the scale range for which a service renders a map. The capabilities specification just includes an optional field with the range of scales for which it is appropriate to generate a map of a layer. Although this information has different semantics (i. e. it refers rather to the data resolution), it might be assumed that the service renders a map at least at these scales. Additionally, as this field is optional, it may happen that some service providers have included this information in free text fields. In this case, a support tool (e.g. for a keyword–based or semantic–enabled search) is necessary, or an additional effort is required from a potential client (e.g. manual visualisation) to estimate the resource utility. Content–based analysis can be useful to extract additional information from a resource, which is not directly provided within its description. The automatic discovery of geospatial resources based on content analysis instead of relying on keywords in the metadata is gaining interest. For instance, Zhang et al. (2010) present a semantic–based application that is able to perform an intelligent content–based search of Web Feature Services (WFS). Although this approach is not directly applicable for non–textual resources such as imagery, image analysis techniques can be used. Image classification and annotation has been extensively investigated in the field of IR, and machine learning approaches have been commonly applied in this area (Baharudin et al., 2007; Gonzalez-Garcia et al., 2007; Sinha and Jain, 2008). These techniques have been also applied to remote sensing imagery for the classification of multispectral imagery over a semi–urban area (Alonso and Malpica, 2008), or to support the update of existing land use databases (Kressler et al., 2005). The heuristics of the method proposed include both techniques for processing the WMS Capabilities document, and techniques for analysing the content of the layers offered by a 58 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES WMS service instance. Finally, it must be noted that any technique for the content–based analysis of a WMS service requires an effective algorithm for collecting a representative set of fragments of an image, instead of analysing the whole extent of an image. The geographic extent defined via an MBBOX is commonly accepted by the geospatial community (including WMS providers) for indexing and coarse resource selection. However, the type of the geographic feature (e. g. a forest, a named place), the spatial object used to represent it (e.g. multipolygons, icons) and the dispersion of spatial data within the dataset have influence on the efficiency of an application that searches for geographic information using a MBBOX. As presented in the previous Section, the semantic annotation of the geographic extent of OWS instances can improve the accuracy in the discovery of WMS services and their layers. Although the WMS services provided as input for the method proposed are only annotated by bounding boxes and it is not always possible to infer a more accurate description of their geographic extent, the algorithm for the collection of image fragments takes into account the variability in the dispersion of data by including a space division strategy. 3.3.2 The method Overview The development of appropriate heuristics requires a prior analysis of the characteristics of the studied phenomenon or process studied. For this purpose, the differences between orthoimage layers and the other layer types have been studied. The lessons learned have been used to develop the deterministic heuristics. A WMS produces spatially referenced maps which are dynamically rendered in a graphical format. The geographic information used for rendering imagery is organised into logical units, i.e. layers, which might be organised as a hierarchical structure. The GetMap operation allows a user to request a single layer, a parent–layer, or any combination of those via explicit indication of layers in the request. According to the content offered by a layer, five general categories can be identified: •Orthoimage Layers. These layers offer orthoimages of high quality such as low altitude aerial photography or satellite orthoimages (normal colour images of high spatial and radiometric resolution) (Schmitt and Stilla, 2010). •Non–orthoimage Satellite Layers. These layers offer non–orthoimage satellite imagery, e.g. radar or laser products. •Coverage Data Layers. These layers offer images produced from coverage data representing continuous phenomena, for example a temperature map produced as an interpolation of sensor measurements (Iosifescu-Enescu et al., 2010), or a coastal ocean model (Schoenhardt et al., 2010). 3.3. SEMANTICS OF THE WMS LAYER 59 Figure 3.6: Overview of the method for the orthoimage layer detection. •Toposheet Layers. These layers offer digitised but not vectorised toposheets, for example scanned maps. •Vector Layers. These layers offer vector datasets, for example country boundaries or icon– based feature representations. Various features of existing WMS services have been taken into account in order to detect the discriminating characteristics of an orthoimage layer. The analysis has considered descriptive features (i.e. the keywords used and declared scale in the Capabilities document), behavioural features (i.e. the presentation scale at which data are visualised by a viewer) and physical features (i. e. the characteristics of the layer image fragments, for example, number of colours). The proposed methodology for layer analysis is a combination of the two different approaches as depicted in Figure 3.6. First, the methodology proposed analyses the metadata contained in the WMS Capabilities document (the MetadataAnalysis). The second part of the method gathers a sample collection of image fragments to explore the content served by the WMS service (the ContentAnalysis). If it is possible to apply the image analysis to the collected samples, the result of the metadata analysis is not considered during the final annotation of the evaluated layer (the LayerAnnotation). Description–based analysis The description–based approach considers the OWS Capabilities document as an information source to identify the keywords that appear with high frequency in the description of orthoimage layers. Although the Capabilities document might be extended by a service provider, only the metadata elements that follow the OGC WMS specification have been considered during this analysis. In the case of the WMS Capabilities document, the offered metadata are split into four groups: •General Service Metadata. This group gathers general service metadata fields (e.g. title, 60 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES abstract,keywords,bbox). •Parental Layer Metadata. This group gathers descriptive elements from the parental path in the hierarchical layer structure (e.g. name,title,abstract,keywords,bbox). •Layer Metadata. This group gathers descriptive elements of the layer which might be inherited from the parental path (e.g. name,title,abstract,keywords,bbox). •Layer Metadata Document. This group contains more precise layer description defined via a link to a standardised metadata document that describes the presented data. It is necessary to identify the deterministic set of reference keywords per each metadata group gthat distinguishes an orthoimage layer. Having a keywords set which is common for a group of orthoimage layers (Koi(g)) and a keywords set which is common for a group of non–orthoimage layers (Knoi(g)), the deterministic set of reference keywords is the difference of Koi(g)and Knoi(g), Koi \Knoi(g)(i.e. the set of all elements of Koi(g)that do not belong to Knoi(g)). The deterministic sets of reference keywords are then used to analyse the metadata of the evaluated layer. Figure 3.7 shows the workflow of the description–based analysis. First, the keywords are extracted per each metadata group from the Capabilities document of the analysed layer (the GeneralServiceMetaKeywExtraction, the ParentalLayerMetaKeywExtraction, the LayerMetaKeywExtraction, the LayerMetaDocKeywExtraction). If any keyword of the extracted set matches with the corresponding reference keywords, the layer is assumed to be an orthoimage layer. Otherwise, it is aNonOrthoimage layer. As for the keywords extraction procedure, it processes separately the metadata groups. First, the XML document is parsed and the text from those fields is extracted and converted into a set of words. The applied tokenisation rules are similar to those that search engines use: (1) the transformation to lowercase, (2) the use of spaces and punctuation as word separators, and (3) the exclusion of words with encoding errors. Digits, one and two letter words, and words commonly ignored by search engines are also excluded, except those with semantic relevance in the geographic domain. Then, only the words of the highest frequency are considered. Content–based analysis Any content–based analysis requires the development of heuristics for effective data collection. In the case of a WMS service, an effective retrieval procedure refers to a process for making requests to the service and collecting a set of representative fragments of the image layer, instead of referring to the full extent of the layer. This set of image fragments allows the development of techniques for content evaluation, in this case, identifying an orthoimage layer via colour–based tests. Figure 3.8 shows the workflow of the content–based analysis. First, the template of the image fragment request (GetMap) is defined using the information obtained from the OWS Capabilities 3.3. SEMANTICS OF THE WMS LAYER 61 Figure 3.7: Workflow of the description–based analysis. document. Then, the collection procedure is invoked. If a valuable image fragment collection has been gathered, the image analysis is performed over the collection. It combines the colour test and the pixel test. The result of the content–based analysis can be Orthoimage or NonOrthoimage according to the result of the image analysis procedure. If the image fragment collection is not valuable, the layer is annotated as NotKnown. The following subsections present the procedure for gathering a valuable image fragment collection (Section 3.3.2), and the details of the proposed heuristics for image analysis (Section 3.3.2). Collection procedure The goal of the collection procedure developed is to gather a set of valuable image fragments from a WMS layer to perform the image analysis. A valuable image fragment is a valid response from a WMS service whose number of colours (NImgC ) is bigger than an established threshold (TNImgC ), i.e. NImgC > TNImgC . In the case of orthoimage layers, a one–colour image fragment is assumed to be non–valuable (TNImgC = 1). It is probable that this condition causes 62 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES Figure 3.8: Workflow of the content–based analysis. erroneous interpretation of an image with vector data, e.g. when an image fragment represents the inside part of a polygon represented by one colour. However, in this work orthoimages are targeted, and the risk of estimating a vector layer as an empty layer (i.e. no positive responses) is acceptable. The collection procedure generates and invokes automatically series of GetMap requests for a layer which is the object of evaluation (the layer name,IDLayer). The definition of the request parameters should ensure homogeneous characteristics of images in the collection, i. e. image type, image size and scale. The image format (FImg), for example JPEG or PNG, cannot differ in order to reduce the variability caused by image generation processes, which could distort the image analysis results. The requested image size, i.e. the number of pixels per height and width (HImg and WImg respectively), should not change either for uniformity of the evaluated collection. As the image size is constant, the scale of a WMS image is determined by the dimensions of the bounding box specified in the request (BBoxr). For every GetMap request a random BBoxrmust be randomly generated within the geographic extent of interest (the work BBox,BBoxw) while preserving the same scale (the request scale) and preventing deformation of representation. 3.3. SEMANTICS OF THE WMS LAYER 63 The scale plays also a significant role in gathering a valuable image fragment collection, as the layer may offer data only for a specific scale range. The request scale should be a presentation scale supported by a layer, i.e. a scale for which the layer offers spatial information (i.e., it produces non–empty images). Therefore, a scale change should be considered if a valuable image fragment is not found at the selected scale. The Capabilities document does not specify a field to declare the presentation scale in the sense it is used here. It allows defining the range of scales for which it is appropriate to generate a map of a layer. Additionally, a provider may not specify this information and the scale recommendation can appear within a free text description of the layer or even in the layer name. In this case, pattern–based searching is used to extract this information. Other issue that should be considered when designing the collection algorithm is the layer geographic extent. The Capabilities document informs about the geographic extent for which the layer offers spatial data, i. e. the bounding box of the layer (BBoxLayer). However, this information might not be accurate enough to get a valuable collection. The BBoxLayer indicates the extent covered by data, but it does not mean that any request within this search area will provide a valuable response. The layer may offer data only for a part of the defined BBoxLayer. For example, especially in the case of vector data, data might be widely dispersed and their relative size might be considerably tiny. This complicates the development of an automatic procedure for collecting valuable images. If the area covered by data is very small compared to the search area, it can be difficult to find any data, and the task collecting images of the whole area at a given scale might not be efficient. Therefore, it is necessary to restrict the number of requests, and if the collecting task has not gathered a valuable collection, the search area is divided into several parts (ND). Then, another series of requests are performed per each new search area to ensure better distribution of spatial requests. This division–requesting task can be repeated if necessary. Figure 3.9 shows a sample layer which requires several division iterations to find a valuable image collection. Although the iteration can be repeated any times, the number of repetitions should depend on the BBoxLayer, because the method is developed to work on a sample collection of image fragments and not on the whole layer geographic extent. The performance aspect must be considered as well, especially for those bounding boxes of layers producing an image collection of considerable dimension (e.g. Europe) at a selected scale. For this reason, in this work, the number of iterations is limited. Algorithm 1 presents the algorithm of the developed collection procedure. It applies the division– requesting task and considers the change of the request scale. Additionally, the algorithm requires the following auxiliary functions to define several thresholds: •ND(i)– a function that defines the number of areas into which the BBoxLayer is split (i. e. the number of BBoxw) in each iteration (i): ND(i) = 4i−1; •NRw(i)– a function that defines the number of requests per each BBoxw(at iteration i): 64 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES Figure 3.9: Example of image division in the collection procedure. NRw(i) =    initialise, if i= 1, for example 100 NRw(i−1)/2,if i > 1; •NImg(i)– a function that defines the expected number of images in the collection after each iteration (if no error occurred): NImg(i) =    ND(i)∗NRw(i),if i= 1 ND(i)∗NRw(i) + NImg (i−1),if i > 1; •TNvImg(i)– a function that defines the threshold to discriminate a valuable image collection (at iteration i), i.e.: 3.3. SEMANTICS OF THE WMS LAYER 71 the presence of more than one geographic feature type. The vector layers that present more than one geographic information logic unit correspond with compound layers, and in practice they should have been split into various related layers. Compound layers have not been considered in this method. Therefore, this kind of single layers are interpreted as noise. No coverage or non–orthoimage satellite layers have been identified within the results. The complete algorithm is characterised by improving the precision from 14.94% to 87.06%, which indicates that the applied image analysis procedure is suitable for the required functionality. On the other hand, the decrease of the recall (from 97.56% to 60.16%) seems to be influenced by the image fragment collection algorithm. Within the negative results there are layers where the algorithm was not able to gather a valuable collection. One of the questions of the collecting process is the dispersion of the requested bounding box within the layer bounding box. The requested bounding boxes are spread more homogeneously during the second and third collecting iterations. However, it has been observed that the number of collected valuable responses increases only in the case that the first iteration retrieves at least one valuable image fragment. 3.3.4 Implementation of the method as a WPS The WMS layer analysis functionality has been published as a Web service in compliance with the OGC WPS specification (Schut, 2007). When a WPS offers a time consuming process, it is necessary to offer an asynchronous communication protocol with the client. Such a WPS usually supports two operations: one operation to invoke the procedure, and another to obtain information on process state and retrieve the results. The analysis of the content of a layer is one of these cases that might take some time. For this reason, the service prototype supports two processes whose execution might be invoked: “analyseLayer” and “getAnalysisResults”. Figure 3.11 shows the class diagram of the implemented prototype. OGCService is an interface that must be implemented by all kinds of OGC services. The OGCWPS interface extends the OGCService interface and offers two methods, the DescribeProcess and Execute, in conformance with the OGC WPS specification. The OGCLayerAnalysisWPS interface extends the OGCWPS functionality to permit users to explicitly invoke the processes offered (“analyseLayer” and “getAnalysisResults”). This interface is implemented by the LayerAnalysisWPS class and the prototype is an instance of this class. As the service is multithread, various analysis might be performed at the same time. The AnalysisProcess class is responsible for the analysis process. The ProcessInfo and RequestInfo classes gather information on the process and the user request, respectively. The AnalysisResult class contains the results of the analysis and information on errors if they have occurred. Some errors may occur while initialising the user request (e.g. “Capabilities document could not be processed.”; “The requested layer < IDLayer > does not exist.”) or during the analysis process (e.g. “Service connection error.”). Figure 3.12 presents the sequence diagram of the communication between a client (the User) and 72 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES Figure 3.11: Class diagram of the implemented WPS. the prototype service (the OGCLayerAnalysisWPS) via the OGCWPS interface. The execution of the “analyseLayer” process invokes the WMS layer analysis (the Execute operation with AnalyseLayer parameter). The process takes a URL of the GetCapabilities request of a WMS service (the URL) as an input parameter. A layer name (the IDLayer) and a user email (the eMail) are optional parameters. If the layer name is provided, only this layer will be analysed. Otherwise, all requestable and single layers found in the Capabilities document will be analysed. The response of this operation is a process identifier (the IDProcess) that the client has to use as a parameter when executing the “getAnalysisResults” process (the Execute operation with GetAnalysisResult parameter). When the analysis process finishes (STATUS=Finished), the results are stored on the server for some period of time (see the note PreserveResults in sequence diagram). The information on this period is provided in the Abstract metadata of the “getAnalysisResults” process description (it may be obtained via the GetCapabilities or DescribeProcess method). It may be configured by the service publisher. By default, it is one week from the time the process finished. If the user email has been provided, an email with the results obtained is sent as well (see sendEmail invocation in sequence diagram). The “getAnalysisResults” response message (in an XML format) has four parts: •Status. The Status block provides information about the current state of the process. The 3.3. SEMANTICS OF THE WMS LAYER 73 Figure 3.12: Sequence diagram of the communication between a client and the OGC WPS for the identification of the orthoimage WMS layer. “Executing” value means that the process is still being performed. The “Finished” status informs that the analysis process has already finished. If ‘NoProcess” value appears, it means that the process of the provided identifier does not exist: either it has been already destroyed or it has never been run. •ProcessInfo. The ProcessInfo element contains the IDProcess element and the CallDateTime element with information on the process invocation moment. •RequestInfo. This block contains information on the user request, i.e. the GetCapabilities request (the wmsURL) and the retrieved response (the CapabilitiesDoc), the layer name (the IDLayer), and the user email (the eMail) if provided. •Response. The Response element gathers the analysis results at the moment of the “getAnalysisResults” request. If the user request has been initialised successfully, the analysis status (the analysisStatus) has “OK” value and the AnalysisResult element summarises the analysis 74 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES results per each analysed layer (the LayerResult). If the analysis status has the “Error” value, the ErrorInfo element is provided instead. The LayerResult element contains the layer name (the IDLayer) and estimated type (the LayerType), and the scale (the Scale) at which it has been analysed. With respect to the process state (Status), the system generates different responses to the “getAnalysisResults” request. This WPS service can be useful when the input of other processes or a potential WMS client needs to access information presented with a certain precision and form. For instance, this WPS service might be a component of an automated catalogue service able to analyse content and extract additional meta–information. 3.4 Application of the methods developed The catalogs that use precise spatial representation instead of MBBOX approximation offer better functionality for the applications based on on–the–fly data integration. An example might be an application which allows displaying geographic information from different OGC services found in a services catalog (see Figure 3.13). The prototype of such enhanced CSW catalogue has been used as the core component of Web application7, which displays spatial data provided from different OGC services. Only services that return responses encoded in GML (i.e. WFS and WCS services) has been considered for assignation of geoindentifiers. GML–encoded response allows unambiguous interpretation in terms of existence of geographic information, which can be confusing in case of an image (e.g. a one colour image might be empty or not). The Web client retrieves OWS services via MBBOX but only enables those which spatial objects overlaps with current bounding box (i.e. the displayed area). Since the list of selectable layers depends on area displayed, it improves user experience. The main disadvantage of this proposal has been the response time of the reference WFS service. To solve this problem we have created a local repository of spatial objects retrieved previously from the reference service. In this way, each instance of service is annotated with its precise spatial object as well. The cache techniques have also improved the services catalog response time and the behaviour of the end application. The method for image layer identification has been applied in the Virtual Spain project, an R&D project supported by the Spanish Government through the Centre for the Development of Industrial Technology whose objective is to define architectures, protocols and standards for an envisioned 3D Internet focusing specially in 3D visualisation, virtual worlds, user interactions and the introduction of semantic capabilities. In particular, one of the experiments proposed in this project is focused on the crawling of services for access and exploitation of images8. The method presented 7http://www.idee.es/IDEE-ServicesSearch/ServicesSearch.html 8http://ev.unizar.es/EV42/ 3.4. APPLICATION OF THE METHODS DEVELOPED 75 Figure 3.13: Services catalog as the support component for application based on on–the–fly data integration. Figure 3.14: Integration of the WPS for orthoimage detection within the Virtual Spain project, an example of generated 3D representations for an urban area (Gran Vía Street, Madrid). has been used in the development of a catalogue of orthoimages9, which are compiled as a result of 9http://ev.unizar.es/EV-ResourcesCatalogueH7/ResourcesCatalogue.html?locale=en 76 CHAPTER 3. CONTENT–BASED SEMANTICS FOR GEOSPATIAL RESOURCES crawling WMS services on the Web. The method, implemented through the OGCLayerAnalysisWPS class as explained in Section 3.3.4, contributes to the filtering of crawled services to detect if they contain image layers. This project also explores how to link these image services with Virtual Globe applications in order to provide a realistic display of Digital Elevation Models (DEMs) for urban areas, which are produced as the result of another experiment in this project. Virtual Globes provide computer–based representations of the real world that are receiving an increasing interest by experts in the geoscience field (Bailey and Chen, 2011). Figure 3.14 depicts the architecture of this workflow. On the one hand, an orthoimage catalogue has been created by filtering orthoimage layers of WMS instances that have been previously gathered from the Web. On the other hand, the generation of DEMs in urban areas has been investigated. Figure 3.14 shows also an example of generation of a 3D model of a real building in the Gran Vía street in Madrid. On the left side, there is a basic model, and on the right side the same building model is combined with an orthoimage requested from the “pnoa” layer (offered by PNOA–IGN WMS service10), which has been selected from the orthoimage catalogue. The image catalog resulted from filtering crawler outcomes can use the additional information about the scale at which the image layer has been identified. This information can be exploited by Web client for recommendation for human users. 3.5 Summary This Chapter has presented two example applications which require additional characterisation of geospatial resources. The approach proposed uses content–based heuristics for dataset sampling which take advantage of standardised interfaces of the Web resources analysed. The first example uses semantic Web technologies to describe OWS. First, the idea of abstraction of a geographic feature from its spatial definition is presented. Then, the current approaches in referencing and identification of geographic features in the Semantic Web and the Geospatial Web are revised. Following the supposition that applying the best practices from the Semantic Web might be useful for the Geospatial Web, here, an administrative geography is created in accordance with Linked Data principles. The principal advantage of using Linked Data technology in geospatial solutions is the possibility of explicit identification of features and abstraction of their spatial definition from footprint and computational representation. The different spatial representation might be accessible via linked instances and chosen according to the application requirements. Such ontology is used as source of geoidentifiers in a geospatial solution and its main advantage lies in using more precise spatial representation and spatial reasoning on the semantic level. A Geoidentifier and corresponding MBBOX used to represent the geographic extents may improve the recall of OGC services catalog. For instance, this improvement has been used in developing Web–based applications that facilitate 10http://www.idee.es/wms/PNOA/PNOA?Request=GetCapabilities&Service=WMS&Version=1.1.1 3.5. SUMMARY 77 on–the–fly data integration. Currently, this early design of the applied ontology has evolved to the Spanish jurisdictional application ontology described in López-Pellicer et al. (2011f). The second part of this Chapter has been dedicated to providing a geoprocessing service for the automatic identification of orthoimages offered through WMS services. In order to demonstrate the feasibility of this geoprocessing service, the implementation has been tested on a collection of WMS instances found on the Web. The human supervision of the experiment results has proved the efficiency of the proposed method (87 % precision and 60 % recall). The image catalog resulted from filtering crawler outcomes has been applied in the Virtual Spain project. Chapter 4 Semantic characterisation of Geospatial Web resources 4.1 Introduction The aim of this Chapter is to propose an architecture of a system dedicated to the automatic creation of geographic metadata of Web resources. Such architecture should be prepared to support various metadata models and different types of Web resources (i.e. to be easily extensible). A heuristic–based method for geographic scope estimation of Web pages has been proposed as well. A prototype, which is able to generate a geographic metadata (in DC profile) of an HTML Web page, has been developed and tested. The metadata model is tailored to the SDI requirements because that is the model used by the metadata catalogues compliant with the OGC Catalogue Service standard. The experiments have been run on a realistic corpus made of OWS Capabilities documents (i.e. generated by a Web crawler focused on OWS (López-Pellicer et al., 2011e)), which has placed the developed prototype in the actual Web environment. These Web pages belong to sites of publishers of Geospatial Web resources; therefore the generated metadata could be used in a catalog. This Chapter is organised as follows. Section 4.2 summarises the existing approaches in automatic creation of metadata from the Web community, with special focus on Web pages and geospatial domain. Section 4.3 introduces an architecture for automatic metadata generation for Web resources. First an abstract workflow of the process necessary to generate metadata is outlined and then, the architecture is briefly described. Section 4.4 presents application of the architecture proposed to generate geographic metadata for the Web pages which are part of Web site of publishers of Geospatial Web resources. Considering that the geographic metadata are hardly provided within Web pages, including those from the geospatial domain, a coverage estimation method has been developed. Then, the prototype implemented and results of performed experiments are described. 79 80 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES In the end, some conclusions are presented. 4.2 Web community and geographic metadata The approaches to the automatic generation of metadata from the geospatial community are appropriate for Geospatial resources; however, they cannot be applied successfully to Web resources such as geoportal Web pages. Therefore, the usage of metadata in Web pages is examined and the main research work from the Web community related to this issue is studied. There is much work done in the field of research on the development and maintenance of metadata of digital Web resources (Ossenbruggen et al., 2004; Nack et al., 2005; Foulonneau and Riley, 2008). Greenberg et al. (2001) shows that non–professionals equal professionals in the creation of metadata for Web resources. However, Web content publishers do not pay attention to asssure proper description of the resources or even deliberately distort it (Golliher, 2008). For example, the content publishers keep on using metadata to try to gain visibility within search engines because the metadata contained within HTML Web pages used to be the base of ranking method. Today, this assumption is erroneous because search engines rank resources mainly with graph–based algorithms (Brin and Page, 1998). The geospatial–based solutions that use Web resources (e.g. HTML Web pages) as part of the searchable content are mainly LBS systems, which are popular in mobile environments. They require generation of some descriptions of the resources and then, indexing them for their further retrieval, and in these terms, they are similar to Web search engines. Although the metadata associated with a Web resource may not be reliable, it may be still be used as the base for the automatic creation of metadata. The header section of an HTML document compliant with the HTML 4 W3C Recommendation (Raggett et al., 1999) may contain metadata via the meta elements (<META>), which contain a property–value pair, i.e. the name (the property name) or http–equiv (the value of header of the HTTP response) and the content (the property value). A schema attribute may be added to specify how to interpret the property value. These values can be eventually described via a metadata profile declared in the head element via a URI. For example, the Dublin Core Metadata Initiative (DCMI) (DCMI, 1995–2012) recommends a DC metadata profile (Johnston and Powell, 2008) that can be encoded using HTML elements and attributes. However, there is no specification that enumerates legal values of the name attribute. The mapping of the metadata used popularly in the Web to a target metadata model might be developed by analysing the W3C and WHATWG recommendations (Hick, 2011; Hickson, 2011) and lists of the meta elements frequently used in Web pages gathered by initiatives such as Metatags.org (Metatags Company Inc., 2012). As for metadata for describing the geographic scope of a Web resource, it can be represented as a disambiguated textual description of a location, a spatial object (e.g. a point or a bounding box) or both. Apart from coverage of the DCMI (and its extensions, e.g. DC.coverage.x) there are other geographic 4.4. APPLYING THE KNOWLEDGE GENERATOR 87 special sovereignty and their principal administrative divisions developed by the United States Federal Government (National Institute of Standards and Technology, 1995); the ISO 3166 Codes for representation of names of countries and their subdivisions (ISO, 2007b); and the Nomenclature of Territorial Units for Statistics (NUTS) developed by the EU (EC, 2003). In this simple ontology, geographic entities are the concepts, and the only relationships of interest are the spatial aggregations, i.e., has–part or part–of. It is a modification of the Administrative Unit domain ontology proposed in López-Pellicer et al. (2008). Additionally, natural phenomena and towns have been considered as well. The resultant ontology gathers geographic entities of the following types: 1. Feature (FT) that represents a natural phenomenon, for example “Danube” (river) and “Alps” (mountains range), 2. Earth region (ERT) that defines international organisations, for example “European Union” and “United Nations”, 3. Country (CT) that represents countries in the world, 4. Region (RT) that represents the top level administrative divisions of a country, 5. Sub–region (SRT) that represents the administrative divisions of a country lower than the top ones, 6. Town (TT) that refers to cities. For example, in case of “Barcelona” toponym, the expected entitygis the “Barcelona” (TT) in “province of Barcelona” (SRT) in “Catalonia” (RT) of “Spain” (CT). The ERT entities are related to countries they gather (has–part), and FT entities are related to countries they belong to (part–of ). The Lge is created by assigning to each item in the Lner the first entitygfrom the ranked list. The geo–scope estimation procedure uses a Lge to calculate frequencies of the geographic entities for different levels of accuracy in the following order: TT,SRT,RT,CT,FT,ERT and EARTH. Each Lge item is represented via the entitygto which it is related at the accuracy level that is being calculated (e.g. “Barcelona” (TT) will be represented by “Catalonia” at RT level of accuracy). The method returns an entitygof maximum frequency and the ESTIMATED code. If the method could not have estimated the coverage (e.g. it fails if the Lge is empty), the “Global” entitygand the ASSIGNED code are returned. The final heuristic (H3+Hhip) is performed as follows. First, the content–based heuristic is run (see Figure 4.4). The gMeta are checked and if the gMeta provide a point, it is used to create the MBBOX and the result code has POINT value. If no spatial object has been distinguished, the text values are analysed to create PngMeta and then the corresponding Lge. If the geo–scope estimation procedure fails (code has ASSIGNED value), a weighted list is created by joining the PngMeta and the Pnmeta (w2(PngMeta), w1(Pnmeta), where wi=i). If the geo–scope estimation procedure fails 88 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES Figure 4.4: Overview of the content–based heuristic. again, the Pnbody is added, new weights are assigned (w3(PngMeta), w2(Pnmeta), w1(Pnbody), where wi=i) and the geo–scope estimation procedure is run again. The host–based heuristic is used only when the heuristic H3fails to estimate the coverage (i.e. it returns ASSIGNED code), which happens usually due to the lack of metadata and poor NER results. The heuristic Hhip tries to extract the ISO country code from host name of the analysed Web page (HName) and if it is not successful, its IP is georeferenced to an ISO country code (HIP ). Then, the ISO code is geocoded to a MBBOX and ESTIMATED code are returned. Table 4.2 shows some examples of Web pages whose coverage has been estimated by the host–based heuristic. The developed content–based heuristic is simple and has several problems. First, the candidate place names are trimmed from their context when using the geocoder. For example, it does not consider other place names from the same Lner, which have been identified near the searched place 4.4. APPLYING THE KNOWLEDGE GENERATOR 89 URL Manual H3HName HIP Hhip H3+Hhip estimation Code (Hhip) (Hhip)Code bnhelp.cz CZ ASSIGNED CZ –CZ ESTIMATED b5m.gipuzkoa Gipuzkoa, Basque ASSIGNED –ES ES ESTIMATED .net Country, ES Table 4.2: Example of the Hhip heuristic results. name within the text. The procedure for the creation of Lge delegates the ranking to the geocoder as well. The algorithm that creates Lge could consider, for example, the re–ranking of geocoding list according to other items within the Lge. The results of the experiments performed shows that this straightforward approach can be satisfactory in the context of this work. 4.4.2 Prototype This section presents the prototype implementation and describes the experiments performed and their results. First, the supported metadata model is introduced with some details on mappings applied. Then, the main characteristics of the implemented prototype are presented. Next, the corpus used and the performed experiments are described, and then the results are discussed. The implemented prototype is dedicated to the generation of geographic metadata of Web pages of Geospatial Web resource providers. The generated metadata will be deployed in an OGC catalogue service. Therefore, the metadata model consists of the core returnable properties supported by the OGC catalogue services (Nebert et al., 2007). Table 4.3 presents a metadata model and the main mappings applied by the system. The system receives an XML document whose schema conforms to this metadata model. The source element contains the URL of the Web page for which the metadata should be generated. The modified (i.e. the metadata creation date), identifier (i.e. a unique identifier of the metadata) and type (i.e. “Web page of a Geospatial Web resource provider.” by default) are filled first. Additionally, the HTML title element is mapped to title and a procedure to extract copyright information from body element is implemented as well. The prototype has been implemented in Java. It supports redirection, i.e. the HTTP redirections and a simple JavaScript declaration to follow a link. The Tika Apache project has been selected as the base for the implementation of the architecture. A generator, which works with the metadata model described above, uses several extractors (e.g. the gMeta extractor and metadata extractor which work on the header element, the title element extractor, coverage estimator). The extractors that extract the Lner lists use the Stanford NER (Finkel et al., 2005). The NER tool is configured in function of the Web page language. Various classifiers have been tested (Manning and Klein, 2003; Faruqui and Padó, 2010). Although the NER tool used in this prototype only handles English or German text properly, it covers 58.3% of the analysed corpus as showed later. By default, the 90 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES CSW Record Model Min..Max HTML Element Mapping contributor 0..* DC.contributor coverage 1..* DC.coverage, coverage, ICBM, geo.position, geo.placename, geo.region, geographic–coverage, DC.coverage.x/ DC.coverage.y, DC.coverage.longitude/ DC.coverage.latitude creator 0..* DC.creator, author, 0..* webauthor modified 1 – description 0..* DC.description, description format 0..* DC.format, content–type (http–equiv) identifier 1 - language 0..* DC.language, language publisher 0..* DC.publisher, publisher relation 0..* DC.relation rights 0..* DC.rights, rights, copyrights source 1 – subject 0..* DC.subject, keywords title 0..* DC.title, application–name (http–equiv) type 1 – Table 4.3: Metadata model and mapping to the HTML meta elements. classifier trained with an English language corpus is selected. The coverage estimator implements the method presented in Section 4.4.1. A geocoding module has been implemented to support the required functionality. It uses the Google Maps API as an external geocoder, which has been chosen due to its global coverage, a rich data model (i.e. it includes administrative divisions up to CT), a multilanguage support and a relevance ranking. It was necessary to provide a geographic ontology with parent–child relations (part–of and has–part) for the other types of geographic features, i.e. feature and earth region. Additionally, an extension for the ISO codes of countries and their subdivisions (ISO, 2007b) has been added because the external geocoder does not support them adequately. Appendix A contains some details of the implemented prototype. Listing 4.1 shows an example of the metadata generated by the prototype. It might be observed that apart from the dc:coverage element some provenance information is offered as well, i.e. the coverage code (gse:code), the coverage in textual format (gse:coverage), the extracted gMeta (gse:covmeta) and the heuristic code (gse:hcode) that informs which heuristic has produced the resultant coverage. The provenance information on metadata generation process is not required by the OGC CSW specification, however it may help in the evaluation of the accuracy of the generated metadata later. 4.4. APPLYING THE KNOWLEDGE GENERATOR 91 4.4.3 Experiment Some experiments have been run to evaluate the functionality of the implemented prototype. The corpus used permits the examination of the system in the conditions of the actual Web environment. Corpus The corpus is a realistic set of resources retrieved from Geospatial Web resources publishers. A set of OWS URLs returned by an OWS crawler (López-Pellicer et al., 2011e) was used to identify the publishers (each URL was trimmed to its host). In this work, it is assumed that it is highly probably that the Web pages reached via this list are pages published by the service publishers. In other words, it is assumed that OWSs are related in some way with the Web pages served by the same host. Listing 4.1: Metadata generated when applying the coverage estimation method. 1<Record > 2<dc:coverage >51.545027 , -0.056262 , 51.545027 , -0.056262 </ dc:coverage > 3<dc:creator>1</dc:creator> 4<dc:creator >London Borough of Hackney </ dc:creator > 5<dct:modified >2011 -12 -14 T22:38:52Z </ dct:modified > 6<dct:abstract >This site has been created by the London Borough of Hackney . The site provides access to information and online requests about the priority services delivered by the council and its partners . </ dct:abstract > 7< dc:format > text / html ; charset =iso -8859 -1 </ dc:format > 8< dc:language >eng </ dc:language > 9<dc:publisher >London Borough of Hackney , Town Hall , Hackney , London E8 1 EA , Tel 020 8 356 5000 , http: // www. hackney .gov . uk </ dc:publisher > 10 < dc:source > http: // www .map . hackney . gov .uk/ LBHackneymap /</ dc:source > 11 <dc:subject >Hackney Map , Hackney , Hackney Council , Borough of Hackney , Hackney Where , HackneyWhere , Property , Map , LLPG , UPRN </ dc:subject > 12 <dc:rights >Copyright London Borough of Hackney , Town Hall , Hackney , London E8 1EA , Tel 020 8 356 5000 , http: // www . hackney .gov . uk [Or this could be a link to a copyright declaration page ]</ dc:rights > 13 < dc:title > Map . Hackney 2.0 </ dc:title > 14 <dc:type > Geospatial Web Resource Provider </ dc:type > 15 <dc:identifier > beb14402 -0018 -4 d20 -997bba7e5176a019 </ dc:identifier > 16 <! -- Provenance Information --> 17 <gse:coverage></gse:coverage> 18 < gse:code > POINT </ gse:code > 19 <gse:hcode>H3</gse:hcode> 20 <gse:covmeta >London Borough of Hackney , London , UK , Global </ gse:covmeta > 21 <gse:covmeta >51.545027 , -0.056262 </ gse:covmeta > 22 <gse:covmeta >Hackney </ gse:covmeta > 23 < gse:covmeta >GB - HCK </ gse:covmeta > 24 <gse:covmeta >51.545027; -0.056262 </ gse:covmeta > 25 </ Record > 92 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES Type Code % Note geoportal Gp 48.09% Geoportal main pages portal P13.45% Portal main pages resource P/ Gp 2.36% A logical part of a portal dedicated to geographic information map visor AV 12.73% Map visor based Web pages or even geoportals other S23.46% Web pages which usually provide demo services or it was difficult to analyse them. They include: 6.18% Companies Web pages. 4.36% Research group Web pages. 4% Entrance point, i. e. pages that require some kind of interaction with user to proceed (e.g. logging or language selection), 2% Personal Web pages 6.73% Other Web pages (mainly community or software pages). Table 4.4: Classification of Web pages in the corpus according to Web site characteristics. Type Code % Coverage local L49% Coverage refers to a part of a country national N30.97% Coverage of a country regional R2.91% Coverage crosses country boundaries global G10.38% Coverage of the Earth out–of–Earth O0.36% Coverage does not refer to the Earth (two examples) notKnown NN 6.38% It was impossible to determine the coverage manually (mainly pages classified as S) Table 4.5: Classification of Web pages in the corpus according to the coverage estimated. The OWS host list (1122 elements) was analysed manually in October 2011 using Chrome browser (version 14). More than half of them (51.1%) were not considered due to some errors (e.g. duplication, connection and page loading errors) or did not provide information that might be processed (e.g. “Under construction”, an empty page, a server test page). The rest of the OWS hosts (549 elements) were analysed in order to identify the geographic scope and the language. Table 4.4 shows classification of the pages according to their origins. It can be observed that 63.9% of Web pages are portal (or geoportal) pages. Some Web sites of map visor–based sites can be classified as geoportals, however, they are treated separately due to the technology used that makes it difficult to analyse their content automatically. The manual estimation of coverage treats a Web site as a whole and considers its published geospatial resources in the estimation. Table 4.5 shows the classification of the corpus according to the coverage estimated. Most pages are in English (43.9%), German (14.4%), Spanish (12%), Polish and Italian (about 4% each), and Czech, French and Catalan (about 3% each). The rest of the examined pages are 4.4. APPLYING THE KNOWLEDGE GENERATOR 93 Metadata element % filled coverage 3.21% title 97.54% subject 43.67% description 42.72% creator 25.71% contributor 1.32% publisher 7.56% rights 10.96% format 80.91% language 34.22% Table 4.6: Summary of metadata extraction. mainly in one of the official languages of Europe but there are examples of Web pages in languages of Asia (e.g. Thai or Chinese). Additionally, there are six pages whose content is in two languages (e.g. atlastenerife.es). The corpus analysis indicates that 87.7% of the sites of the regional or global geographic scope (14 and 50 pages respectively) use the English language. This feature is in accordance with common sense when targeting an international audience, a language used at global–level is preferable (or at least language used in the geographic scope, for example, Spanish language seems to be adequate to the region of Latin America). On the other hand, the internationalisation of Web pages (i.e. availability in other languages) has not been evaluated. Therefore it should be understood that the language attribute of a page refers to the language in which the Web page is offered by default. Experiments and result The experiment consisted in generating the geographic metadata for each Web page from the corpus. Due to the dynamic characteristics of the Web (temporal unavailability of the Web resources), several test runs have been performed during the period of November and December of 2011. In general, 3.1% of the elements of the corpus were not processed due to some errors (e.g. data format errors, frequently repeated connexion problem). Table 4.6 summarises the extracted metadata by applying the defined mapping. The generated metadata will be exploited in a catalogue. Therefore, the faulty metadata have been filtered out (i.e. metadata without title, description or subject fields). After removing metadata that do not conform to this restriction (2.4%), all remaining elements do have at least title, description and subject fields. It can be observed that the information about the geographic scope (coverage) is rare in the examined corpus, and it varies in format (i.e. textual information in different format, lat/long point). Therefore, the next experiment consisted in applying the coverage estimator to obtain a geographic scope (i.e. MBBOX). The Web pages for which it was not possible to estimate coverage manually (6.4%) have been removed from the corpus as well. Table 4.7 summarises the percentage of removed elements of corpus. The results of the coverage estimation experiment (see Table 4.8) shows that H3produces equal 94 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES Lang Total Process Not valuable Coverage Processed Analysed error metadata NN EN 241 11 6 29 195 DE 79 3 2 1 73 Other 229 3 5 5 216 Total 549 17 13 35 484 (100%) (3.10%) (2.37%) (6.38%) (88.16%) Table 4.7: Trimmed corpus (Lang–language, NN – coverage not estimated). results to the manual estimation of the geographic scope of Web pages in almost half of cases. 73.4% of results are correct with the country accuracy or better (i.e. the country of the computed entityg is equal to the country of the manually estimated entityg). In other words, the H3procedure yields acceptable results in 73%, and the erroneous results in 26.6%. After applying the Hhip the coverage estimator produced acceptable results in 78.9%. The result is poor due to several problems. The NER tool is not properly configured for almost half of the corpus. Therefore, the H3produces poor results for them. Surprisingly, the H3produces a similar percentage of errors for the EN corpus. Closer analysis of the corpus and the H3results have shown that H3behaves better for pages classified as national or local than for those classified as global, regional or out–of–Earth. Since 87.7% pages of those classified as global,regional or out–of–Earth consist of Web pages in English, therefore, the results are worse than expected for the EN corpus. An improvement of the equal results after applying the Hhip should not be expected because most of the elements of the corpus are classified as local and the Hhip handles only the country level coverage or higher. Nevertheless, the result shows in fact an improvement, i.e. the number of errors decreases and the percentage of the acceptable results increases. This tendency is not shown in the EN language part of the corpus. After meticulous analysis of the results it has been observed that this effect is produced by the fact that 72.3% of the elements evaluated by the Hhip do not permit the estimation of the ISO code from the host name. In such a case, the Hhip georeferences IP and introduces an error. The coverage estimator is quite good for the DE language part of the corpus. Therefore, it might be suspected that if the NER tool is properly configured, the coverage estimator is efficient (for Web pages classified as local and national) for the DE language part of the corpus at least. Nevertheless, the estimator should be improved for the Web pages classified as global, regional and other. 4.4.4 Evaluation of the content–based heuristic As mentioned before, the functionality of NER tool and its configuration influence the results of the coverage estimation method. In this work, the method uses a geocoder module to geocode the NER results. It removes items that do not match any geographic entity of a valuable type (e.g., a street should be removed because it is not considered). In practice, a NER item can be: 4.4. APPLYING THE KNOWLEDGE GENERATOR 95 EN DE Other Total Total 195 73 216 484 H3137 45 89 55.99% (% of Total) (70.26%) (61.64%) (41.20%) H3Acceptable 97 41 61 73.43% (% of H3) (70.80%) (91.11%) (68.54%) H3Equal 66 32 33 48.34% (% of H3) (48.18%) (71.11%) (37.08%) H3Error 40 4 28 26.57% (% of H3) (29.20%) (8.89%) (31.46%) H3+Hhip 195 73 216 100% (% of Total) (100%) (100%) (100%) H3+Hhip Acceptable 134 67 181 78.93% (% of Total) (68.72%) (91.78%) (83.80%) H3+Hhip Equal 81 39 81 41.53% (% of Total) (41.54%) (53.43%) (37.50%) H3+Hhip Error 61 6 35 21.07% (% of Total) (31.28%) (8.22%) (16.20%) Table 4.8: Results of the experiment on coverage estimation. •Error. It is a false positive that is not a place name at all (i.e., there is no place of such a name). The NLP techniques analyses language structure and may produce some errors. These values are usually geocoded to a street and, in this way, can be removed. •Mistake. It is a false positive which is recognised as a toponym due to ambiguity problem (i.e., it is a toponym in another context). The elimination of these false positives depends on the type of geocoded entity. •Hit. It is a positive result which is a toponym in the used context. Such toponym might be eliminated or eventually geocoded wrongly, if its type is ignored by the system (e.g. it is a street name). In the case of toponym elimination, it is an expected behaviour. However, the geocoding errors are not desired. The result of the method can also be affected by the relevance ranking of the geocoder applied. If a text is geocoded successfully, it produces a ranked list of geocoded toponyms. In the case of the geocoder used in this work, the list size depends on ambiguity of the toponym, and it varies between 1 and 15 items usually. The first item, whose type is valuable, is returned. The geocoder may introduce false positives, for example, a street name can be geocoded to a town. Additionally, any errors and lacks in a reference dataset of the used geocoder may influence the final results. The efficiency of a GIR system is usually measured via precision and recall. In the case of a NER tool, for example, the first one is measured as the number of toponyms returned by the NER tool divided by the total number of items in the NER list. As for the recall, it is measured as the number of toponyms returned by the NER tool divided by the total number of toponyms within the 96 CHAPTER 4. SEMANTIC CHARACTERISATION OF GEOSPATIAL WEB RESOURCES Text Language NER Tool Configuration PG−RPCEP EN EN 86.10% 78.31% DE DE 74.49% 69.78% ES EN 81.52% 67.20% SE EN 63.64% 66.87% PL EN 90.91% 72.73% DE EN 50.90% 48.34% Table 4.9: Evaluation of the coverage estimation method. analysed text. In this work the precision of the coverage estimation method (PCEP ) is evaluated. The geocoder–relative precision (G–R precision,PG−R), a modified precision metric, is measured as the number of items of the NER list that have been geocoded successfully divided by the total size of the list, i.e.: PG−R=GT(Lner ) S(Lner ) GT(list)is a geocoding function that considers only the entitygof type T; T∈(ET, FT, ERT, CT, RT, SRT, TT); S(list)is a function to calculate the size of a list. This metric is adjusted to the proposed approach and it indicates the procentage of the NER list that will be used in the coverage estimation method. Although it may introduce some errors due to ambiguity problems of toponyms and the ranking applied, it can be calculated automatically for any text. Evaluation of the system precision requires manual analysis of the parsed text. Therefore, it has been done for a sample of documents from the corpus. Each toponym generated by the method has been validated with text in order to check if it has been geocoded to the right entity. In general, the better NER and its configuration, the better functionality of the coverage estimator will be achieved. As expected, the improvement of NER tool functionality is observed when using the language dedicated classifier. Table 4.9 presents the experiment results performed over the corpus when considering text language and the configuration of the Stanford NER tool (i.e. language dedicated classifiers). It can be observed that the NER results are better when it is configured adequately to the German language. Additionally, the NER trained for English language performs with similar G–R precision for English and Spanish text. For example, in the case of the Web pages from the Polish language corpus, surprisingly the value of PG−Ris even higher than for English (90.9%). The PCEP decreases in almost all cases but it maintains around 70% (except EN language configuration for German text). 5.3. SEARCHING VIA A GENERAL SEARCH ENGINE 103 Navigational goal In the case of the navigational goal, the user needs to find a service or Web page that he has in mind. For example, the ornithologist might want to find the home page of a concrete provider to see if any other services are offered as well. Informational goal The user also may expect a ranked list of OWS resources and Web pages related to some topic (informational). For example, the ornithologist can make an exploratory search of desertification maps and related pages, or he might gather OWS for building a thematic collection of services about climate components of the Iberian Peninsula. Transactional goal If the user search goal is transactional the user wants to learn the technical details of an OWS service. For example, the ornithologist wants to be able to test a WMS service. These domain–specific search goals might be achieved with help of a general search engine. However, knowledge about its functionality and resource characteristics is required to develop proper search strategies. 5.3.2 Search strategy In the scenario of OWS discovery through a general search engine, users make use of the fact that geoportals often publish documents with hyper–links to them, usually through catalogue viewers (Maguire and Longley, 2005). Therefore search engine crawlers can access and index service descriptions. However, the query result varies in function of the strategy. Two main strategies might be developed in general: basic and expert strategy, which depend on the user’s ability to work with search engines. Basic strategy In the basic strategy the terms in query are those found in the URL of OWS descriptions along with terms related to the required information. For example, the search terms ’getcapabilities coastal’ will return a list of Web resources that contains those terms, such as an HTML page (e.g. a service description) with a hyper–link that invokes the getCapabilites operation. Expert strategy An expert strategy refines the basic strategy (Bartley, 2005; Sample et al., 2006), as search engines usually allow users to define constraints on the URL of the target resource. In this way, the query 104 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE Figure 5.1: Three kinds of search activities and exploratory search (source: Marchionini (2006)). may be restricted to the documents whose URLs match the pattern of requests for the OWS service description. For example, ’inurl:getcapabilities coastal’request in case of Google SE returns the Capabilities documents as the response to the getCapabilities operation. Users who use general search engines to search for geospatial resources might be practitioners of GIS tools but not necessarily specialists in geographic information and related technology. Therefore they will have to investigate the proper way to exploit general search engines for their purposes (i.e. to look thorough a SE documentation and OWS characteristics). However, having the possibility to use a tool that supports the non–expert users in searching for geospatial resources would save their time and effort. 5.4 Non–expert user support As shown in previous section, discovery and searching for geospatial resources on the Web requires expert knowledge. This is because searching for OWS instances has a mainly transactional character. Therefore, a domain–oriented SE that applies special techniques may support the users out of the GI community. For example, a search layer over a general SE that applies the expert search strategy (i.e. via automatised extension of user requests with specialised operators) might be one of the possible solutions. Additionally, a search client should be designed in the manner needed to support user goals other than transactional. Humans have poor short–term memory, i.e. they have a limited ability to search and interpret textual and/or tabular data (Miller, 1956). Therefore, a crucial aspect of achieving success in implementing such systems for their reuse is its acceptance. In this scope, the system’s user interface is critical, as even minor usability problems will demotivate users (Seaman et al., 2003). There are many important theoretical models of information search (e.g. a stratified model 5.4. NON–EXPERT USER SUPPORT 105 presented in Saracevic (1997) which summarises Belkin’s and Ingrewsen’s). In the context of this work, exploratory search seems to be the best choice because it has been identified as especially appropriate for users that lack domain–specific knowledge. Such a search system helps users to explore, overcome uncertainty, and learn, rather than just providing search results (White et al., 2006; Marchionini, 2006) (see Figure 5.1). Exploratory search can be supported in Web search engines by using categorised overviews of Web search results (Kules, 2006), based on meaningful and stable categories. This approach “can provide substantial benefits when searchers need to explore, understand, and assess their results. When information needs are evolving or imprecise, categorized overviews can stimulate relevant ideas, provoke illuminating questions, and guide searchers to useful information they might not otherwise find. When searchers need to gather information from multiple perspectives or sources, categorized overviews can make those aspects visible for interactive filtering and exploration.” (Kules, 2006) One of the existing, successful approaches to search interface that support exploratory search is faceted search and browsing. It is popularly used by search application for searching within DLs in diverse of domains, such as e–commerce (e.g., Ebay5, Amazon6) or scientific results (e.g. ScienceDirect7, PubMed8). Faceted search interface is successful as it usually supports and encourages two aspects of user search behaviour: browsing (i.e. exploration) and querying (in particular query refinement). 5.4.1 Faceted classification Facets have been used by librarians and information scientists to structure information for a long time already. The Classification Research Group indicated faceted classification to be the basis for all IR in 1955 (Broughton, 2006), and then, also librarians started using facet analysis (Ranganathan, 2006). Facets are defined as “a set of meaningful labels organized in such a way as to reflect the concepts relevant to a domain” (Hearst, 2006), and represent “the categories, properties, attributes, characteristics, relations, functions or concepts that are central to the set of documents or entities being organized and which are of particular interest to the user group” (La Barre, 2007). Ranganathan (1967) states that faceted classification “of a particular universe [of entities] is made on the basis of characteristics”. Following La Barre (2010), “(...) characteristic [of entities] is equivalent to a mathematical parameter presenting a range of possible factors, aspects, or elements that assist in the identification of a collection of distinct cases. In facet theory, each parameter creates a dimension, or small number of groupings, and each grouping represents a facet.” Therefore, each facet has potentially multiple dimensions (Ranganathan, 1967). 5http://www.ebay.com/ 6http://www.amazon.co.uk/ 7http://www.sciencedirect.com/ 8http://www.ncbi.nlm.nih.gov/ 106 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE Not each classification system is a faceted classification (Ranganathan identifies eight kinds of classification systems). There are three important issues which distinguish faceted classification (Vickery, 1966): •Faceted classification system is created by strict application of rules and facet analysis such that: “every distinctive logical category should be isolated, every characteristic should be clearly formulated, and each new relation should be recognized.” •Facets are not locked into “rigid enumerative schedules, but are left free to combine with each other in fullest freedom, so that every type of relation between terms and between subjects may be expressed.” •Faceted classification system “breaks free from the restriction of traditional classification to the hierarchical, genus–species relations. By combining terms in compound subjects, it introduces new logical relations between them, thus better reflecting the complexity of knowledge.” 5.4.2 Faceted search and browsing Faceted classification provides a usable method which permits browsing information collections via multiple categories simultaneously (Hearst, 2000; Hearst et al., 2002). According to Hearst (2009) “[i]n a properly designed faceted navigation interface, the user can browse the information collection from any of the different facets as a starting point, and after starting with one facet, can then navigate using any other facet.” Fagan (2010) provides summary of empirical evidence related to faceted browsing found in the information science literature as follows: •Facets are useful for creating navigation structures. •Faceted categorisation greatly facilitates efficient retrieval in database searching. •Facets help avoid dead ends. •Users are faster when using a faceted system. •Success in finding relevant results is higher with a faceted system. •Users find more results with a faceted system. •Users also seem to like facets, although they do not always immediately have a positive reaction. •Users prefer search results organised into predictable, multidimensional hierarchies. •Participants’ satisfaction is higher with a faceted system. 5.4. NON–EXPERT USER SUPPORT 107 Additionally, Fagan performs an empirical study on the usability of search systems; the study shows that faceted browsing improves user performance. Fagan (2010) argues that all these results build a solid case for including facets in search interfaces. Kules proposes a set of principles for the design of exploratory search interfaces in his dissertation (Kules, 2006): •Provide overviews of large sets of results. •Organise overviews around meaningful categories. •Clarify and visualise category structure. •Tightly couple category labels to result list. •Ensure that the full category information is available. •Support multiple types of categories and visual presentations. •Use separate facets for each type of category. •Arrange text for scanning/ skimming. •Visually encode quantitative attributes on a stable visual structure. This set of principles is “useful for digital library and Web search designers, information architects, and Web developers because they provide guidance for the appropriate integration of visual overviews with search result lists, and particularly for the textual surrogates embedded in result lists. These principles embed a strong call for the surfacing of structure – which is often used internally by search engines, but less often exposed at the user interface – without abandoning the tried and true value of text” (Kules, 2006). Therefore, a search system that uses faceted classification should be designed and developed with this set of principles in mind, e.g. several meaningful categories should be used, the category structure should be clarified and visualised, and search result lists should be tightly coupled with category labels. Other works in this context cover technical aspects of the faceted search. Koren et al. (2008), for example, indicates that a faceted search engine essentially performs two separate retrieval tasks: (1) user query matching documents, and (2) recommended query refinement. The first one has been intensively investigated within the IR research community. The faceted classification features of the category isolation and ability to combine facets enables query refinement in a straightforward manner. It is based on the usage of refinement operators. Values of such specialised operators are mapped to facet dimensions which allow grouping of resources via categories. In general, two approaches to query refinement, (1) exclusion and (2) limitation, have been distinguished after analysing some examples of popular online applications that offer faceted search (e.g. Amazon, ScienceDirect). Exclusion allows removal from the result list of those elements whose 108 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE category belongs to one of the defined values. Limitation trims the result list to those elements which are categorised as one of the selected values. This system behaviour is usually offered via interaction with GUI, for example by series of check–boxes on a category values. The refinement operators can also be offered by search interface, i.e. they can be used as search operators. There are free–text search operators and restricted search operators.Free–text search operators are those search operators whose values can be explicitly specified by a user. Values of the restricted search operators have to belong to a controlled set. Such an operator might appear in an advanced search GUI with a controlled list or check–box list. In this way, only values from the controlled set are used as values of the restricted search operators within the search request. This restriction on the input values eliminates user errors and limits possible options which improve system performance. Some works indicate that, even providing the advanced search capacity, the simple keywords search should be still available for a user. If simple search is enabled (i.e. there is one search box to input a request), usually only the free–text search operators are processed as part of a user request. They might be allowed with some restrictions as well, for example, without the possibility of using multiple values or complex logical queries (i.e. “AND”, “OR”). The refinement operators can also be part of the navigation interface to enable browsing of resources via a category. For example, a check–box can also be a link to a list of resources grouped via the specified category. These operators usually correspond to the restricted search operators. The coherent navigation interface and the resultant category–based lists of resources exposed for their further exploration (i.e. with associated links to description of related resources) are the basis of the exploratory search. A search system which offers faceted search and browsing fulfils one of the principles for Web search interfaces provided by Rose (2006), i.e. “the interface should support the iterative nature of the search task. In particular, it should invite refinement and exploration.” Additionally, according to Rose “different interfaces (or at least different forms of interaction) should be available to match different search goals”. Therefore, the system proposed should be adjusted to search for OWS resource and related information. In particular faceted search and browsing supports the informational search goal because it helps with the gathering of information on a particular subject. However, the transactional character of search for OWS instances demands some specific facilities, for example, ability to interact with OWS instances. 5.5 Domain–specific search user interface The design requirements of the Search User Interface (SUI) that exploits facets invlove important issues when designing a search system, in particular, coherence of search, browsing and navigational interaction model. Additionally, the domain–specific characteristics of searched resources should be considered when defining the set of facets and interactions with the system. Examples of faceted search and browsing systems can be found on the Web, including online portals that help users to 5.5. DOMAIN–SPECIFIC SEARCH USER INTERFACE 109 search for Web resources relevant to the scope of this work. Therefore, two main communities have been considered as those of major relevance to this work, the Web service and the Geospatial Web community. First, some relevant Web search clients from related areas are examined to identify their main characteristics. Then, Section 5.6.1 discusses the proposed facets, refinement operators and the implementation decisions which have been taken. 5.5.1 Web service community The important elements that influence the usability of search system are the SE interface and the search client design. Mohammed et al. (2006) describe a UDDI–based SE offered as a desktop application. The SE search for medical Web services that appear on the Web. According to the authors, the developed SVG Search Engine client (i.e. a UDDI client) allows discovery and usage of the required Web services. The search criteria are (1) the service name, (2) the thematic classification (i.e. hierarchical classification schemes of Breast Cancer), and (3) the service groups. The work focuses on the security aspects of such system, and it has been developed as a desktop application. There is no reference to an existing tool; therefore, it is difficult to evaluate functionality of the developed interface. A good reference on a Web service search portal might be seekda’s Web Services portal9, a result of several research projects in which seekda10 has been involved. The main projects are the Service– Finder and the Service Detective. The Service–Finder project11, (funded by the 7th Framework Programme for Research and Development of the European Commission and finalised in December 2009) has been dedicated to “develop a platform for service discovery in which Web Services are embedded in a Web 2.0 environment”12. Brockmans et al. (2008) and Brockmans et al. (2009) discuss the requirements and architecture of such a system. These requirements refer to system components (i.e. service search interface, service related information and user community interface), external resources (i.e. provider and service related information sources), and service description standards (i.e. WSDL documents in the first place and RESTful APIs as the next step). The proposed architecture distinguishes following components: •Ontologies (i.e. Generic Service–Finder Ontology and Service Categories) •Service Crawler •Automatic Annotator •Conceptual Indexer and Matcher •Service–Finder Portal 9http://webservices.seekda.com/ 10http://www.seekda.com/ 11http://www.service-finder.eu/ 12http://www.seekda.com/en/research/projects/service-finder 110 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE •Cluster Engine •Service–Finder Interface (Search Interface, Navigation Interface) Service Detective project13, funded by FIT–IT14, that “create an architecture for a Web Service SE that automatically aggregates information from heterogeneous sources to facilitate discovery of both WSDL and RESTful services”15. The project has been dedicated to the provision of a new approach to Web service ranking (Steinmetz and Lausen, 2009) (e.g. the description that is available on the Web, their hyperlink relations, monitoring information, etc.), a focused crawler that gathers SOAP Web Services (i.e. their WSDL documents), the RESTful Web Services (i.e. the online documents on their APIs, and a WSDL document if provided), and related documents. The system aggregates Web service annotations (Steinmetz et al., 2009) and stores meta–information in RDF triples. The ontologies used have partly been developed in the scope of the Service–Finder project (i.e. Service– Finder Ontology16 and Crawl Ontology17). The seekda’s Web Services portal uses a Web service repository which contains 28.606 services offered by 7.739 providers18. The services have been gathered via a focused crawler that searches the Web for the valid WSDL documents or the links submitted manually by providers. This portal favours creation of user communities. It encourages users to create an account as it is the only way to add the free–form annotations (i.e. tags) or wiki–like comments about services and providers, and to rate services and provider. Each service is described via a country, its provider, a URL of its WSDL document and its cached version (i.e. an XML or HTML document), age, the type of server, a category according the documentation found within the WSDL document (i.e. “none”, “partial” or “good”), a textual description (from the WSDL document, or added by the provider and/or users), and user rating and tags. The portal also offers some monitoring information about services (i.e. availability) and the possibility of invoking the operation declared in the associated WSDL document. Each provider is annotated with a country (via IP georeferencing), some textual descriptions, a list of tags and the homepage URL. A list of offered services is available as well. As for SE interface, it processes requests with restrictions on the country, the provider and free–form tags. The ranking might be modified according to relevance, available documentation, availability, provider, country and age. The first 250 results are presented for each query (there is information on the total number of matching results). The client search interface offers both kinds of search form, a simple form (i.e. the one field form) and an advanced search form (one field per each restriction operator allowed). Although the exploratory navigation is intuitive and coherent, only simple logic requests are supported. The system does not support full faceted search which requires combination of multiple logic restrictions 13http://service-detective.sti2.at/ 14http://www.fit-it.at/ 15http://www.seekda.com/en/research/projects/service-detective 16http://www.service-finder.eu/ontologies/ServiceOntology 17http://seekda.com/ontologies/CrawlOntology 18Last accessed: 09/02/2012 5.5. DOMAIN–SPECIFIC SEARCH USER INTERFACE 111 on declared operators. Currently, there is on–going work on a new version of the portal which includes support to RESTful Web Services and information on mashups as well. 5.5.2 Geospatial community A catalogue service plays a fundamental role in an SDI as it allows the reusage of resources, and it is a central element of a system for the metadata management and discovery. A geoportal is an effective solution for such a system (Baldini et al., 2010), as it is accessible from a Web browser. Currently, there is a variety of configurable frameworks, and it is possible to develop geoportals in an easy and fast manner. For example, GeoNetwork19 is open–source software designed to improve the accessibility of a wide variety of data and the associated metadata. It implements the Portal and Catalog SDI components as defined in the OGC Reference Architecture. GeoServer20 is another example of such software. TerraViva! GeoServer21 is a geoportal that helps in finding interactive maps, GIS datasets, satellite imagery and related applications. Following the OGC GP–RA, geoportals can offer an interoperable search across different catalogues (Giuliani et al., 2011). Such system is a gateway that queries one or more registered catalogues. The GEOSS project is a good example of such an interoperable system that can query multiple catalogues registered in its system. GEOSS contributors use the Component and Service Registry to register their geospatial services for their further discovery through Clearinghouse and invocation by GEO Portal. The central element of the GEO Portal22is a map viewer which is used to show recent updates of resources. The main browsing menu allows users to explore resources according to the SBAs (two–level categorisation schema) and then, to refine the result by the location (a controlled list of regions). Simple search (i.e. free keywords search) and mentioned browsing return resources categorised via their type with indication of number of resources which fall into each category. The top–level categories are: Dataset, Monitoring and Observation Systems, Computational Model, Initiatives, Web sites and documents, Data Services, Software and Applications and Others. A resource in the result list is described via part of the abstract, links to related Web resources (documents, Web pages or Visors) and link to resource description tab. The resource description tab has two possible views, (1) “Summary”, a default view which shows selected metadata (Contact Details, Abstract, Organisation Name, Distributor, Date Stamp and related documents) and (2) “Full Description” which returns full metadata record (encoded in HTML) from the system catalogue. The refinement of search results refers to (1) location (as (1.1) bbox, (1.2) region name from a list or (1.3) region from an interactive map), (2) additional keywords, and (3) social benefit area (one or more category from two–level classification schema). In the context of search for OWS services, identification of OWS services might be a bit confusing in the beginning. As for user search goals, the informational 19http://geonetwork-opensource.org/ 20http://geoserver.org/display/GEOS/Welcome 21http://geoserver.isciences.com:8080/geonetwork/srv/en/main.home 22http://www.geoportal.org/web/guest/geo_home 112 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE goal is quite well supported. A topical query returns a variety of resources (not only OWS services) and the returned OWS services can have associated related documents. The navigational goal (e.g. looking for homepage of a service provider) might be a bit difficult due to an overwhelming number of results for some queries. Accomplishment of the transactional goal seems to be a bit demanding for a non–expert user. Although there is a link to the Capabilities document in the main result list it is automatically redirected to the map visor23. The request point of OWS service can be found after navigating to the “Full Description” As described in Bai et al. (2009), the resources are manually registered. Resource incorporation into an SDI is characterised by a top–down control which involves metadata curation. Such approach is not effective in systems based on Web crawling. Growing interest in crawling the Web for the Geospatial Web resources (López-Pellicer et al., 2011e; Li et al., 2010) and non–SDI resources as an alternative source of valuable information (Craglia et al., 2008; Keßler and Bishr, 2009) give bases to expect that OWS service portals similar to the seekda’s portal will become popular soon. 5.6 Geospatial Web Search Engine The proposed GWSE exploits a general Web SE (i.e. a remote SE) to search for geospatial resources. Focused crawlers are used to identify OWS calls via patterns within the results of the remote SE. In this work, the crawlers are developed to detect the most popular OWS specifications. The OWS resources discovered are classified into service description (the Capabilities document), item description (the response to the Describe* operations) and item (the content published by the service). Table 5.1 summarises the description of the main operations. As for the Search User Interface (SUI), the proposed search system offers to the user the ability to define a search strategy (none,basic,expert), and the search client has been developed in order to help support the three search goals (i.e. navigational,informational and transactional as introduced in Section 5.3.1) in the context of the search for OWS resources. Focused crawlers applied are not levered here because this topic is out of scope of this work (please refer to López-Pellicer et al. (2011e) and Li et al. (2010) for details of those systems). The following Section presents the offered search, browsing and navigational interface. Next, the developed search strategies and process of indexing of OWS content (the OWS Crawler) are presented, and then the role of search operators in USI design is outlined. Finally, the architecture and implementation of a prototype is briefly described. 5.6.1 Facets for search, browsing and navigational interface In a faceted search and browsing system, three types of interactions with USI have to be considered, i.e. search, browsing and navigation. In the context of this work, these concepts are understood as 23There are some usability issues which make it difficult to evaluate this approach. 5.6. GEOSPATIAL WEB SEARCH ENGINE 119 (i.e. they have been used by a crawler in order to identify the OWS resource). The others are Web pages of the OWS publisher. These Web pages have been generated from the domain URL as described in Chapter 4. There is also a link to visualise the resource in a map. The full description of a resource is accessible via the “title” link in the summary description. Each resource is described via information from the Capabilities document which has been extracted using a metadata crosswalk (Nogueras-Iso et al., 2004). In the case of a service, user has the ability to invoke the descriptive operations or to retrieve some items. The requestable elements are described via information from the Capabilities document and the result of descriptive operation is encoded in HTML. Also, some summaries of dataset are shown if it is possible to create them. The discriminative elements are identified. The values from low variable fields are listed with quantitative attributes (i.e. the number of items in a dataset which satisfy the restriction), and an appropriate request is generated and enabled via a link. Theoretical evaluation The design principles proposed in Kules (2006) help in development of systems that support non– expert users in exploring the search results. Table 5.4 describes the manner in which the design principles are accomplished by the GWSE. Id Design Principle Application 1Provide overviews of large sets of The first 50 results are available. The full number results. of matching resources is showed also. 2Organise overviews around meaningful Several stables classification schemes are used. categories. 3Visualise and clarify category The check–box and browsing tabs are organised structure. according to the schemes’ structure. 4Tightly couple category labels to The check–boxes indicate which categories are result list. currently applied. 5Ensure that full category information Not applicable. is available. 6Support multiple types of categories Not supported. and visual presentations. 7Use separate facets for each type of Supported. category. 8Arrange text for scanning/skimming. The result list contains descriptional snippets. 9Visually encode quantitative attributes The refinement menu and browsing tabs on a stable visual structure. provide quantitative attributes. Table 5.4: Accoplishment of the design principles for exploratory search interfaces. 120 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE Seven of the nine principles are fully supported. The fifth design principle is not applicable in this work because no taxonomy of deep hierarchy has been used. In this context, the wide vocabularies can be the obstacle. Domain and Provider are horizontally extensive. Anyway, the browsing menu allows users to browse through them. The sixth design principle is not accomplished because the user cannot define their own categories and no only single presentation style of search results is enabled. As for user search goals, faceted search interface, along with the information model, can support the informational goal especially well. The informational goal can be reached via the exploratory search, as it enables search and browsing collections of resources using multiple categories. Also, the association of non–OWS Web resources (e.g. provider Web site) extends the information scope that can be explored by users. The navigational goal can be achieved, for example, via the “site” operator or by the navigational interface (e.g. by navigating to the provider Web site). The enablement of interaction with OWS services offers extended assistance to the transactional goal. A test module has been created, similar to that one which is offered by seekda’ portal for invoking the operations declared within a WSDL document. It generates a set of modifiable requests in order to allow users to interact with the resource. In this work, the set of request–response fields are only generated for WFS, WCS and WMS services. 5.6.3 OWS search and indexing procedure When the user invokes a search query, two parallel search processes are run, i.e. the task of querying the remote SE (RemoteSE) and the task of querying the local repositories (Repository). The FastWebCrawling is performed over the results from a remote SE when the user query is being handled, while the long–time MainOWSCrawling is performed in the idle time of the system. Figure 5.5 shows an overview of the searching and indexing tasks that are performed by the developed system. The following Sections present some details of these processes. Search procedure When search query is invoked, two parallel search processes are run, i.e. the remote SE is queried and the local repository is exploited. According to the selected search strategy, the user query may be extended automatically with additional tags to improve remote SE efficiency. Then, the fast Web crawling is performed over the SE response. The fast Web crawling consists in the identification of OWS operation calls among resources obtained from the remote SE. The service descriptions of identified OWS instances are indexed and integrated with results from the local repository to produce the user response. The remote SE response is saved for the main OWS crawling process which is invoked apart for performance reason. This process applies the Web Crawler and the OWS Crawler (detailed in Section 5.6.3) to generate the local repository content, i.e. to identify OWS resources in the Web, 5.6. GEOSPATIAL WEB SEARCH ENGINE 121 Figure 5.5: Overview of the searching and indexing tasks performed by Geospatial Web Search Engine. retrieve and index OWS content. The Web Crawler returns OWS resource URLs with a “parent” link associated (i.e., a URL of the Web document where the OWS resource has been identified). This contextual information is necessary to supply the informational search goal. If the OWS resource URL is the item description or item, for example a GetFeature operation call, the automatically created service description URL of the identified service instance will be also associated with the parent resource with the GetFeature operation URL. In this way, the local repository gathers resources obtained from the past requests and the system searchable content is growing continuously. The search strategy selected influences the search task. The none strategy deactivates the search in the local repository (OWS descriptions, OWS content and parent information), and the returned OWS resources are those identified within the remote SE response. The only difference between the basic and expert strategies is the manner of extending the user query before passing it to the remote SE, i.e. with “getcapabilities” or “inurl:getcapabilities”, respectively. Additionally, when a spatial search is used, it is removed from the request that is passed to the remote SE. The responses from the remote SE are crawled by Web robots (with a depth factor of 1) which identify and extract OWS resource candidate list for future content retrieval and indexing. The 122 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE service description links of the identified OWS resources state the Remote Candidate List (RCL). If a spatial restriction is used, the geographical extent defined within the Capabilities documents is checked. At the same time, the search through local repositories produces the Local Candidate List of OWS resources (LCL). While searching without any spatial restrictions, only the text indexes are exploited. First, with the consideration of “site” and “inurl” operators if defined, the text index is used to identify those OWS resource links that match the searching terms. It produces a scored list of OWS candidates. The final OWS list is ordered according to the OWS relevance calculated from the matching score. In the case of a spatial search, the spatial restriction is applied first: the OWS description repository is searched for all candidates that provide data from the requested area (with consideration of “site” and “inurl” operators if defined). Then, the term–based search described above is applied for the obtained list of OWS candidates. The type of the spatial restriction used determines how the OWS candidate list is retrieved. When a point is defined, only this geometry is used in the spatial search. In the case of the “location” operation, the toponym is transformed into a list of geometry candidates using a gazetteer, and the user has to chose one of them. The geometry is applied for OWS search. When both spatial restrictions are used, first the “location” operator is considered. The results from the RCL and LCL are merged. The RCL resources that appear in the LCL are removed and the rest of the elements are indexed and scored (the content retrieval is postponed). Then, one result list is created from RCL and LCL according to the obtained score and the links to the OWS resources are annotated with links to the Web pages, where the resources have been found. Indexing OWS resources OWS service can be seen as an entrance point to the content of the invisible Web. Ru and Horowitz (2005) have identified challenges which are raised when such content is being indexed: 1. The lack of knowledge of the underlying database schema. Therefore, it is difficult to generate the form assignments that generate information–rich resulting pages. 2. A variety of interfaces of the invisible Web sites (even in the same domain). Therefore it is hard to design a universal form–filling method. 3. The volume of the information in the invisible Web. In the case of OWS resources, there is a set of known interfaces guided by OGC specifications which focus on technical aspects and detail interfaces and encodings. These interfaces are self–describing by providing the Capabilities documents. Additionally, the services which expose content of potential 5.6. GEOSPATIAL WEB SEARCH ENGINE 123 Service/Resource Service Description Element Description Item WMS GetCapabilities WCS GetCapabilities DescribeCoverage WPS GetCapabilities DescribeProcess WFS GetCapabilities DescribeFeatureType GetFeature CSW GetCapabilities GetRecords Table 5.5: OWS resources of potential interest to be indexed by the Geospatial Web Search Engine. interest to be indexed (e.g. WFS), provide data model scheme, and it allows the development of heuristics for automatic procedures. A specialised module called OWS crawler has been developed, which is responsible for indexing the OWS resources. It requests OWS services whose Capabilities documents (i.e., the Service Description) have been stored in the temporary repository (i.e. a repository which gathers the Capabilities documents identified within response of the remote SE). A type of OWS service provides some hints on operations that return the valuable content for indexing. Table 5.5 shows the OWS resources (per an OWS type) that may be considered by the indexing system, and the operation that retrieves them. For example, the Capabilities document of any OWS service should be indexed. In the case of WCS, WPS or WFS, the descriptive documents of the requestable elements (i.e. the Element Description) should be also retrieved and indexed, and in the case of the WFS service, the GetFature operation allows the system to index the served content (i.e. the Items). It is possible to determine the number of requests necessary to be performed in order to index the descriptive documents published by an OWS service (i.e. one GetCapabilities call per each service and one Describe* call per each requestable element defined within the Capabilities document). The text–indexing function ignores all XML tags, and all nodes that define geometries or contain numeric data. OWS resources are also indexed spatially via a bounding box defined within the descriptive documents. Indexing the whole content offered by a WFS service is not efficient. A more appropriate approach can be the creation and indexing of some data summaries, (e.g. a data distribution histogram used in the database community (Ioannidis, 2003)). In this work, the data summaries are created by identifying the discriminative elements within the data model, i.e. the low variable fields and high variable fields. For example, in the case of a gazetteer service, the field that contains an entity name will be a high variable field and the field which contains the entity category will be a low variable field. The low variable fields are the classifying fields. The unique values of these fields can be extracted and indexed. Then, the histograms of these fields can be created. The high variable fields are used by a content retrieval procedure to create histograms. The content retrieval procedure is developed by exploiting the OGC Filter Encoding Standard (Vretanos, 2010a), that allows restrictions to be added on the returned items (or features). 124 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE It defines an XML encoding for filter expressions that logically combines the constraints on the properties of an item in order to identify a particular subset of items to be operated upon. For example, it is possible to identify a subset of items by using constraints specified on values of spatial, temporal and/or scalar properties. First, several test requests are performed and textual fields are distinguished. The fields with the discriminative elements are identified via additional validation requests. For example, the values of alow variable field change less frequently than the values of the other fields. Therefore, restrictions on this discriminative element in the future retrieval requests (i.e. with values already gathered) should return items distinct from those already retrieved. Appendix C presents some details of the method developed for creating and indexing data summaries of the content published by WFS services. An additional issue is the periodical update of the remote content which is considered by any centralised system based on indexing remote content. In this work, it is assumed that the content published by the remote services is relatively static. The procedures for periodical update should focus on the Capabilities documents, and the indexing of the service resources should be repeated only if the service description changes. 5.6.4 Architecture and Implementation The overview of the architecture proposed for the GWSE system is shown in Figure 5.6. The RequestManager is responsible for the support of the user search. The input user request is parsed, which consists of the query validation, cleaning (i.e. trimming and steaming) if it is necessary, the query extension (according to chosen strategy), and then the internal query model is initialised. The SE adapter (the RemoteSEAdapter) maps this initialised model into the first–level SE interface and dispatches the query created. The response is stored in the local repository (the TMPRep) for future crawling. In a parallel manner, the fast crawling is performed by the Web Crawler to generate the RCL as described in Section 5.6.3. The OWSrSearcher is responsible for generation of the LCL, which then is integrated with the RCL by the RequestManager. Finally, the FacetController manages the presentation of the results according to selected facets. In the idle system time, the responses gathered in the TMPRep are crawled (the Web Crawler) and the OWS resource retrieval and indexing is performed (the OWS Crawler). The system prototype uses Google WebSearch API25 as the remote general SE. The “location:<LOC>” operator is removed from the search request and the toponym (i.e. “<LOC>”) is added as an additional term. The libraries under open source licenses have been used for implementation: indexing and textual searches are supported by using Lucene26, and Hibernate27 is used to communicate with 25http://code.google.com/apis/websearch 26http://lucene.apache.org/ 27http://www.hibernate.org/ 5.7. SYSTEM EVALUATION 125 Figure 5.6: Overview of the architecture of the Geospatial Web Search Engine. PostgreSQL28 database. These technologies provide support to the required spatial functionality. 5.7 System Evaluation This Section presents two experiments performed in order to evaluate the prototype developed in this work. The first one is an automated test that compares the result precision of the GWSE prototype with the precision of a selected general SE (i.e. Google Web Search API). The second experiment is dedicated to a human evaluation of the SUI design proposed. The participants have been asked to perform a search task and fill an online questionnaire. 5.7.1 Precision testing Several test have been performed to compare the response precision of different search strategies for different search goals. The search goals have been defined for purpose of this work according to the characteristics of the searched resources, and Google Web Search API has been used as the remote SE. The NOMGEO database29 of georeferenced toponyms of Spain has been used to create the local gazetteer for the experiment purpose. This restriction conditioned the way of “location” operation testing, i.e. only place names from Spain could be used. As the none strategy deactivates geospatial functionality, the results of this strategy have not been considered. The system gathers information during its work; therefore the evaluation has been performed at the moment when there were about one hundred services indexed in the local repository. The top ten results have been considered for precision estimation, following the common approach from studies on the SE evaluation which usually uses ten or twenty first results (Griesbaum, 2004; Kumar et al., 2005; MacFarlane, 2007; Tawileh et al., 2010). First, the precision of the Google Web Search Engine has been estimated by evaluating the results from the RCL. The precision has been calculated separately for each search goal. In the case of the 28http://www.postgresql.org/ 29http://www.idee.es/IDEE-WFS-Nomenclator-NG/services 126 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE Strategy/Goal list interact discovery expert 98% 94.6% 94.7% basic 68% 18% 76% Table 5.6: Geospatial search precision for different combination of search strategies with search goals. interaction search goal a hit is the link to a service Capabilities document. The list search goal is satisfied by a service Capabilities document or a page with a link to it. In the case of the discovery search goal, the hits from the list search goal are extended with all pages which allow a client to find a service Capabilities document when navigating from them with a maximum of three hops. The evaluation queries have been divided into non–spatial queries, spatial queries with a point, and spatial queries with the “location” operator. Each of the sets consists of 15 queries. Six test runs have been performed (October 2011) to evaluate precision results when combining search strategies with search goals. While searching without spatial restriction, the expert strategy precision was high for all search goals, and the basic strategy precision was the worst for the interaction search goal (see Table 5.6). Independently from the search goal, the spatial search with a point usually produced an empty result. Analysis of results has shown that the point is treated as text and the engine returns only the textual matches. When using the “location” operator, the results produced have the same precision as those obtained when searching without spatial restriction. This behaviour is correct because the value of the spatial operator is added as an additional term when calling the remote SE. In the second stage of the experiment, the system was always successful (100%) as the system operated on an initialised service repository. As for system performance, the response time increased significantly in the second stage of the experiment, however, this experiment was not focused on this aspect of the system. 5.7.2 Interface Evaluation Utility and usability of a system design are the key quality attributes which determine whether the system is useful (Nielsen, 2003). Design utility indicates whether it provides the features aimed, and the second one refers to those properties of the interface that determine how easy it is to use. Nielsen (2003) identifies five components of usability: •Learnability. This quality component informs how easy is it for users to accomplish basic tasks the first time they encounter the interface. •Efficiency. This quality component assesses how quickly users can accomplish their tasks after they learn how to use the interface. 5.7. SYSTEM EVALUATION 127 •Memorability. This quality component assesses how long it takes users to reestablish proficiency after a period of non–use. •Errors. This quality component assesses how many errors users make, how severe are these errors, and how easy is it for users to recover from these errors. •Satisfaction. This quality component assesses how pleasant or satisfying is it to use the interface. One of the popular approaches to the evaluation of UI usability is the discount usability testing (Nielsen, 1989), which involves having a small set of evaluators who examines the interface and judges its compliance with recognised usability principles. The SUI design presented in this work focuses on the information model for search and browsing of Geospatial Web resources. Therefore, design utility should be the target of this evaluation. The principal goal of the SUI design developed is the support to non–expert users, i.e. the users that lack specific knowledge on geographic resources. Therefore, experts in the geospatial domain and non–expert users should be able to perform a search task successfully. Additionally, the interface design applied favours exploratory search which should breed an effect of learning about geospatial resources. As for evaluation of usability, it might be expected that following known and successful approaches from the Web should assure system usability. Here, the successful execution of a search task by a user will be considered as a positive rate of learnability and efficiency in some degree. User satisfaction will be considered in this work as well. An online questionnaire has been created to evaluate the GWSE prototype, and a group of Web users have been invited via emails to participate in it. Since the target participants should consist of expert and non–expert users, some efforts have been made to encourage researchers in the Geospatial domain and out of it to participate. The survey has three parts as follows: •Pre–search questionnaire. This part of survey gathers some general information on participants (e.g., age, gender, country, education level, etc.). It also allows the evaluation the user’s prior knowledge on geographic information resources (in order to classify the user as an expert or non–expert), and a potential interest in these resources. •Search–task questionnaire. The participants are asked to perform a search for geographic information resources. The required task involves the three potential search goals (i.e., informational,transactional,navigational). The search task has to be performed by using the prototype application (GWSE) or the Google Search Engine (GoogleSE). The type of the SE is assigned in order to maintain the proportion between the SE used (i.e. GWSE and GoogleSE users) and also considering the classification of the user. •Post–search questionnaire. This part of survey allows participants to express their opinion on the tool used and satisfaction with the task result. It also attempts to estimate the learning 128 CHAPTER 5. GEOSPATIAL WEB SEARCH ENGINE User Perfil/Search Engine Expert Non–expert Completion Error Completion Error GoogleSE 6474 GWSE 7573 Table 5.7: Classification of the survey participants. Completion and Error columns indicate the number of questionnaires which have been completed properly or not, respectively. Search Task Result/Search Engine Success Failure Expert Non–expert Expert Non–expert GoogleSE 5 (83.3%) 0 (0%) 1 (16.7%) 7 (100%) GWSE 7 (100%) 5 (71.4%) 0 (0%) 2 (28.6%) Table 5.8: The survey results. effect of the exploratory search. The search system developed is characterised by a focus on the Geospatial domain with strong bias towards OWS resources. Therefore, only some aspects of the recommendations on exploratory search task design provided in Kules and Capra (2008) have been employed. A participant has been asked to imagine himself/herself to be a researcher who is going to write a report on the fresh water supplies in the Mediterranean region, including information on droughts and floods if it is possible. This introduction also includes a brief overview of targeted resources (i.e. what is a Geospatial Web service and types of OWS services). The search task which should be performed consists in gathering necessary information for the report: (1) some Web sites which publish geospatial resources on the subject should be found, (2) the information resources should not be in a form of a map only (i.e. some examples of resources of processable data should be given), and published via services are preferable. The services which offer the most useful information should be identified among the found services and the reasons for the selection should be given. The homepages of providers of the selected services should be identified if possible as well. The task has been restricted by time limit (i.e. 20 minutes). At the end, the test participants have been asked to rate each search engine in terms of the intervals: Terrible–Wonderful, Frustrating–Satisfying, Dull–Stimulating, Confusing– Clear and Rigid–Flexible, which are based on the Questionnaire for User Interface Satisfaction (QUIS) for the subjective evaluation measures (Chin et al., 1988). Additionally, the participants were asked to evaluate the increase of their knowledge about Geospatial Web resources. The survey ran in January 2012. In total, 43 participants have been invited to participate. Only the well–completed questionnaires have been considered and the questionnaires which have not been finished (e.g. abandoned) have been removed from the further analysis. Table 5.7 summarises characteristics of the participants according to the declared knowledge on geospatial resources and