Adding Retrieval Augmented Generation to the MOSAIC Framework

ADDING RETRIEVAL AUGMENTED GENERATION TO THE MOSAIC FRAMEWORK F. Holz1, D. Scharf1, A. Nussbaumer1, S. Gürtl1 1Graz University of Technology, Graz, Austria Abstract This paper presents a concept for adding RetrievalAugmented Generation (RAG) features to the MOSAIC framework. MOSAIC enables web search in segments of the Open Web Index (OWI), in order to establish a specialpurpose search engine. An extension, MOSAIC-RAG, has been developed that adopts a RAG approach. It is designed as a modular framework that has integrated a set of processing modules built on generative AI models, such as a module for re-ranking the search result, a module for summarising the full texts of the search result, or a module for summarising all search results. These modules can be ordered in an arbitrary sequence, in order to configure an overall process to improve the search result. Such configurations can be adapted for specific purposes and saved for later reuse. INTRODUCTION Recently, Large Language Models (LLM) have become very popular, because humans can interact with them in natural language when requesting information. They are capable of generating texts in various contexts, such as answering questions, providing extensive information, or summarising texts. In contrast to traditional search engines, they do not deliver original web documents, but generate responses based on a vast amount of information that has been used to train them. Though this type of information searching might be attractive for many people, there are also problems such as the phenomenon of hallucinations, outdated information, and missing information sources. Retrieval-Augmented Generation (RAG) seeks to combine LLMs with traditional search engines. Different techniques have been proposed explaining how search engines are enriched with LLM functionalities[1]. A simple technique consists of the use of text chunks retrieved from a search engine for feeding and prompting an LLM. More advanced features include the improvement of the search query, as well as the re-ranking or summarisation of the results with the help of an LLM. Such an integration has several advantages and partially overcomes the aforementioned problems of LLMs. A web index with current data can inject up-todate information into LLMs, and also provide original web documents on demand. Thus, hallucination is mitigated by providing factual knowledge in combination with generated texts. This paper presents a RAG approach that is based on the Open Web Index (OWI). A special-purpose search engine created with data from the OWI is integrated with a framework that processes the retrieved data using different kinds of AI models. The next section describes the overall concept of this framework. This section is followed by a more detailed description of the modules used to improve the search process. Finally, an application is presented that showcases how a RAG system can be set up with our approach. CONCEPT AND MODULAR FRAMEWORK The overall aim of MOSAIC-RAG 1 is to enrich search engines using the Open Web Index (OWI) with features provided by Large Language Models (LLMs). The enrichment is mainly performed by further processing the search result, such as providing summarisations, re-rankings, or conversational search. The result is delivered to the end-user via a built-in web interface or an API that can be used by external applications. The overall concept is depicted in Figure 1 and explained in more detail in this section. OWI Index Slice MOSAIC MOSAIC-RAG Web Interface Application Download Import Search and Retrieve Module Module Module API API Chatbot ChromaDB API Search and Retrieve Figure 1: Conceptual design of MOSAIC-RAG. The first step of creating a MOSAIC-RAG application consists in the creation of an index slice that serves as the underlying database for the search engine. Index slices are smallor medium-sized indices containing web documents related to a certain topic or a particular purpose. More precisely, they contain an inverted index represented in CIFF format 2 and metadata of each web document represented in Parquet format 3 . The metadata include the title, full text, 1https://opencode.it4i.eu/openwebsearcheu-public/mosaic-rag 2https://github.com/osirrc/ciff 3https://parquet.apache.org/ https://doi.org/10.5281/zenodo.17209496 URL, language, geo-coordinates, topic, and other information of the web document. Such slices can be downloaded from the OWI using queries that specify the domain and content of the index slice [2]. For example, index slices can contain web documents related to a certain topic, such as science news, a specific language, such as Finnish, or are part of a certain top-level domain. The second step consists of the preparation of the search engine that delivers search results using the index slice. There are two options that are compatible with MOSAICRAG. First, MOSAIC is a framework and generic search application that makes index slices searchable [3]. Second, Chroma 4 is a vector database that allows to search documents using vector embeddings. Both search engines provide an API that allows the search for web documents and delivers lists of web documents including their metadata and full text. Ingesting data slices works different for each of these search engines. MOSAIC is designed to easily integrate index slices by just copying them into a resource directory. Each index slice is represented as an index in MOSAIC and can be searched individually. Importing index slices into Chroma needs some pre-processing, as it requires vector embeddings for each web document, that can be created with suitable models, such as the Jina Embeddings 2 Model[4]. In Chroma, each web document is represented as a triple consisting of an ID, the vector embedding, and the metadata from the Parquet file. Later the search query is also represented as vector embedding using the same model, which allows Chroma to retrieve matching documents. In the future, the vector embedding will also be part of the OWI, which simplifies the importing procedure. The core of MOSAIC-RAG is a modular pipeline that enriches the search result retrieved from the search engine. It includes a suite of processing modules that can perform various transformations of the search result. The currently available modules are described in the next section. For example, the full text of each result item (web document) can be summarised, the list of result items can be re-ranked, or an overall summary can be created out of the search result. The set of currently available modules is extensible and new modules can be added by implementing a base class that manges a data frame consisting of the search result. The rows of the data frame consist of the individual web documents and the columns comprise their metadata. Each module can manipulate the data frame in any way. Typically, a row with newly calculated information is added, for example with summarisation of the full text or by computing a new ranking (see Fig 2). Each module that uses an LLM to process the data can either chose to run the LLM locally, i.e., directly from the Python code, or use a remote inference point. The remote inference point can be configured globally for the whole MOSAIC-RAG instance. For this purpose, either a LiteLLM5or OpenAI compatible endpoint is required. 4https://www.trychroma.com/ 5https://www.litellm.ai Module 1: Data Source 231 Title 1 1 Text 1 424 Title 2 2 Text 2 352 Title 3 3 Text 3 453 Title 4 4 Text 3 Module 2: Summarisation Module 3: Re-ranker ID Title Rank Full text 231 Title 1 1 Text 1 Sum 1 424 Title 2 2 Text 2 Sum 2 352 Title 3 3 Text 3 Sum 3 453 Title 4 4 Text 3 Sum 4 ID Title Rank Full text Sum. 231 Title 1 1 Text 1 Sum 1 3 424 Title 2 2 Text 2 Sum 2 2 352 Title 3 3 Text 3 Sum 3 4 453 Title 4 4 Text 3 Sum 4 1 ID Title Rank Full text Sum. Re-rank Data from Search Engine Figure 2: Modular Pipeline with data frames The modules can be sequenced in any order depending on the purpose how the results should be processed. Thus a user can create a certain sequence of processing modules, in order to define the behaviour of MOSAIC-RAG (see also next section). Such a configuration is ephemeral, only lasting for the duration of the browser session. However, MOSAIC-RAG provides two ways of loading and saving the full configuration, i.e., custom color theme, custom titles, and the pipeline configuration. First, configuration can be downloaded in JSON format. Users can upload this JSON file to the frontend to restore a saved configuration. Second, this configuration can be stored on the server under a unique ID to be retrieved using a custom URL. Each module has also a few parameters to steer their behaviour, such as the selection which LLM should be used for the summarisation. In addition to the modular pipeline, MOSAIC-RAG also supports a conversional search functionality. In a chat box, the user can ask an LLM questions about the current set of search results. The conversational search agent is instructed to only give answers based on the actual search results, not based on its own world knowledge. In order to interact with MOSAIC-RAG, a web interface is provided that enables both the search and the configuration of the modular pipeline. The web interface uses MOSAICRAGS fully documented API. This allows other applications to use the full functionalities of the service. RAG MODULES This section describes the 18 currently implemented pipeline modules. These modules are organised in five groups, depending on their functionality: data source, sumhttps://doi.org/10.5281/zenodo.17209496 marisation, re-ranking, pre-processing, and metadata analysis. The data source modules deal with retrieving search results from external search engines when a user starts a query. Currently two search engines are supported, MOSAIC and Chroma. Details can be configured, such as the index used by MOSAIC or the embedding model used by Chroma. Furthermore, the number of search results can be limited. The data source module converts the data gathered in those search engines into a dataframe. This dataframe gets passed through the configured pipeline modules sequentially. After the final module, the dataframe gets sent to the user according to the API specification. Multiple data source modules can also be added to the same pipeline, allowing for the aggregation of data from different sources (e.g. multiple MOSAIC instances). The pre-processing modules mainly deal with text cleaning and organising of the result set. There are modules to remove HTML tags and stop words, or to perform stemming operation on the text. These functions might not be needed in every case, as the search results may already be cleaned by the original search engine. As computing power is often limited, the Reduction Module is important because it reduces the size of the internal data frame based on a condition (usually the ranking). When processing large result sets in a pipeline containing at least one LLM module, such as a LLM Summarizer or an Embedder, the execution time of the total pipeline can be greatly reduced by decreasing the number of processed documents. Therefore, after performing some re-ranking, it might be sufficient to keep the best few documents and discard the rest. The re-ranking modules change the ranking of the result set. Currently, four re-ranking modules are implemented by default in MOSIAC-RAG. The embedding re-ranker performs a new ranking based on the similarity of embedding vectors. Those embedding vectors will be created using the SentenceTransformer Python library if they do not already exist in the data frame. The TF-IDF (term frequency - inverse document frequency) [5] re-ranker is among the simplest and fastest approaches, allowing documents to be re-ranked based on their TF-IDF vector representations and a chosen similarity metric. Currently, MOSAIC-RAG supports the similarity metrics Euclidean distance, Manhattan distance, cosine similarity, and BM25. The latter differs slightly from the others, as it does not rely on the full TF-IDF vector representation. A BM25 ranking algorithm is also used by MOSAIC for its search. The two other re-ranking modules are based on the principle of large-language-modelre-ranking [6]. Here large language models (LLMs) are used to identify which document fits the given query best in a set of given candidate documents. The Group-Style LLM re-ranker module ranks documents by comparing a set of candidate documents against a given query and allowing the LLM to determine which document best matches the query. For each comparison, a score is assigned to the document that fits best. This process is repeated across all possible document combinations, given both the size of the candidate set and the total number of documents [7, 8]. The language model and the size of the candidate set can be configured. The final pre-implemented re-ranking module is the Tournament-Style LLM Re-ranker. Like the Group-Style variant, it relies on an LLM for re-ranking, but it reduces the number of required document comparisons, the most time-consuming step, by leveraging an existing ranking and refining it locally. The process follows the structure of a tournament tree, where the winning document advances while the losing one is eliminated. This approach requires significantly fewer LLM comparisons, improving efficiency. However, it functions more as a ranking enhancement than a full re-ranking. As it depends heavily on the initial ranking used as the seed, its effectiveness is greatest for identifying the top-ranked documents relevant to a query, while ranking quality tends to diminish further down the list. It is important to note that all pre-implemented re-ranking modules operate solely on the documents retrieved in the initial stage and do not perform any additional retrieval themselves [9]. There are two types of summarisation modules. The first one summarises the full text of each web document in the result set, while the second one generates one summary of all the documents in the result set. In both cases an LLM with targeted prompts is employed for these tasks. Finally there are three metadata analysis modules. The first one is a simple word counter that counts the number of words in a web document. The Sentiment Analyser calculates a sentiment score for each web document. Based on six output scores for each sentiment the highest score is taken and stored in the dataframe. The relevance marking module marks parts of the full text that are most relevant. Both the sentiment analysing module and the relevance marking module use an LLM for their task. APPLICATION CASE For demonstrating how a RAG system can be set up and configured with MOSAIC-RAG, an application has been created that enables search in the domain of arts. This arts search engine is depicted in Fig. 3. First, an index slice has been created that only includes web documents related to arts. This was achieved by selecting web documents in the Open Web Index that are tagged with the Curlie label Arts. After integrating this index slice in MOSAIC, the service is started. Second, MOSAIC-RAG is set up to act as an arts search engine. Hence, a MOSAIC-RAG data source module is configured to use the data from the arts index of the previously started MOSAIC service. Then an Embedding Re-ranking Module is added to improve the search result. Finally, a summarisation module is added that provides an overall summary of the search result on top. Finally, the appearance of the web interface is configured. The title is changed to Arts Search and the colour scheme is set to dark-orange. The whole configuration is saved and an ID is automatically created, which allows to share this configuration via a single URL. https://doi.org/10.5281/zenodo.17209496 Figure 3: The Web Interface of MOSAIC-RAG configured as arts search engine. The summarisation and search results are on the left side and the processing pipeline on the right side. CONCLUSION AND OUTLOOK The main contribution of this paper consists of a RetrievalAugmented Generation approach in the context of the Open Web Index. A special purpose and vertical search engine created with data from the OWI is integrated with a framework that processes the retrieved data using different kinds of LLMs. Future work will include user studies that investigate the usefulness and acceptance of this approach. Furthermore, different configurations will be created and tested, in order to better understand their benefits for the user. In particular, the benefit for end-users of summarisations and re-rankings will be investigated. ACKNOWLEDGEMENTS This work has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070014 (OpenWebSearch.EU, https://doi.org/10.3030/101070014). REFERENCES [1] Y. Gao et al.,Retrieval-augmented generation for large language models: A survey, 2024. https://arxiv.org/abs/ 2312.10997 [2] M. Granitzer et al., “Impact and development of an open web index for open web search,” Journal of the Association for Information Science and Technology, 2023. 10.1002/asi. 24818 [3] S. Gürtl, MOSAIC: Empowering a Modular Framework for Configurable and Tailored Web Search based on an Open Web Index, 2024. https://diglib.tugraz.at/ diplomaTheses [4] M. Günther et al.,Jina embeddings 2: 8192-token generalpurpose text embeddings for long documents, 2023. [5] K. Sparck Jones, “A statistical interpretation of term specificity and its application in retrieval,” Journal of documentation, vol. 28, no. 1, pp. 11–21, 1972. [6] Y. Zhu et al., “Large language models for information retrieval: A survey,” ACM Transactions on Information Systems, 2025. 10.1145/3748304 [7] W. Sun et al.,Is chatgpt good at search? investigating large language models as re-ranking agents, 2024. https://arxiv. org/abs/2304.09542 [8] J. Sun, X. Zhong, S. Zhou, and J. Han, “Dynamicrag: Leveraging outputs of large language model as feedback for dynamic reranking in retrieval-augmented generation,” arXiv preprint arXiv:2505.07233, 2025. [9] M. Rathee, S. MacAvaney, and A. Anand, “Guiding retrieval using llm-based listwise rankers,” in European Conference on Information Retrieval, Springer, 2025, pp. 230–246. https://doi.org/10.5281/zenodo.17209496