Using The Open Web Index To Create New Search Applications For Research.fi

USING THE OPEN WEB INDEX TO CREATE NEW SEARCH APPLICATIONS FOR RESEARCH.FI J.Theodoropoulos∗, J. Kesäniemi, CSC – IT Center for Science Abstract We plan to utilize the Open Web Index to enhance the features of Research.fi, a Finnish portal showcasing national research related outputs. Our plans include improvements to the existing search functionality of the website and the introduction of fields for related publications. Our work aims to improve the accessibility of information related to research and provide a generalizable framework for similar open search projects. EXTENDED ABSTRACT OpenWebSearch.eu (OWS) is a project funded by Horizon Europe with the aim of building the European Open Web Index (OWI). The OWI provides an option for Internet search that is sovereign from the large technological companies dominating the field. While not targeting to build a single search engine to compete with the existing ones, the OWI provides a backbone to new search applications including search engine verticals. [1] Research.fi is a Finnish service offered by the Ministry of Education and Culture, and developed by the CSC - IT Center for Science, also an participating organisation in OWS. The service contains information on research conducted in Finland including publications, grants, organizations and infrastructures. Information available in Research.fi is based on the National Research Information Hub, which acts as a national aggregator of research-related data in Finland. The purpose of the Research.fi is to improve the discovery of Finnish research, support the reuse of research-related information, and provide profiles for researchers for sharing their activities and outputs. [2] In the first phase of the project, a prototype of the science index, i.e. a search vertical focusing on science related content, was created. First, crawled content was downloaded and filtered using the owilix tool [3] to include content only in the three languages supported by the Research.fi portal: Finnish, Swedish and English. To create a dataset produced in a consistent manner, only data crawled in 2025 were used. Pages related to science were further filtered using the corresponding Curlie labels [4] provided in the metadata of the crawled websites. An additional need to filter out pages with low quality texts was noticed in initial testing, and a language model [7] was used to identify these pages for removal. These low quality pages included old message board threads but also pages from universities’ websites that did not contain text that could be scraped in an efficient manner. From the curated set of pages, the index was built with tools from OWS project [5], and MOSAIC [6] was utilized ∗[email protected] for queries. Although still in progress, multiple possible applications have been identified for using the curated science index as part of the Research.fi portal. • Expanding the search results of Research.fi with sites from the OWI, thus containing information of science and research conducted outside of Finland. • Creating a field of related content for entries present in Research.fi; e.g. using the OWI to identify research done using a specific grant. • Utilizing the OWI to measure and provide insights of the impact of research output in metrics other than citations. • Using a knowledge graph and topic modeling based on data from Research.fi to enhance queries made to the OWI. • Leveraging the capabilities of large language models with a retrieval-augmented generation based search implementation. This project is under continuous development with the goal to implement new features to the Research.fi portal by the end of this year. To maintain reliable features with data from the OWI, development is carried out in a way that enables reproduction of the results and updating them with more data from daily crawls. ACKNOWLEDGEMENTS The authors are employed by CSC – IT Center for Science and their work is partially funded by OpenWebSearch.eu. REFERENCES [1] OpenWebSearch.eu, https://openwebsearch.eu [2] Research.fi, https://research.fi [3] owilix, https://opencode.it4i.eu/ openwebsearcheu-public/owi-cli [4] Curlie.org, https://curlie.org [5] Spark indexer, https://opencode.it4i.eu/ openwebsearcheu-public/spark-indexer [6] MOSAIC, https://opencode.it4i.eu/ openwebsearcheu-public/mosaic [7] llm-data-textbook-quality-fasttext-classifierv2, https://huggingface.co/kenhktsui/ llm-data-textbook-quality-fasttext-classifier-v2 https://doi.org/10.5281/zenodo.17238350