Querying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation

Querying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation Explorando las Profundidades: Revelando las Fortalezas y Desaf´ıos de los Modelos de Lenguaje de Gran Escala en la Generaci´on de SPARQL Adri´an Ghajari, Salvador Ros, ´ Alvaro P´erez Spanish National University of Distance Education agha[email protected], {sros, alvaro.perez}@scc.uned.es Abstract: The emergence of the Semantic Web has precipitated a proliferation of structured data manifested in the form of knowledge graphs, underscoring the imperative of natural language interfaces to enhance accessibility to these repositories of information. The capacity to articulate queries in natural language and subsequently retrieve data through SPARQL queries assumes paramount importance. In the present investigation, we have scrutinized the efficacy of in-context learning based on an agent-based architecture in facilitating the construction of SPARQL queries. Contrary to initial expectations, the augmentation of in-context learning prompts through agent-based mechanisms has been found to diminish the efficacy of Language Model-based Systems (LLMS), as it is perceived as extraneous ”noise,” thereby delineating the constraints inherent in this approach. The results highlight the need to delve deeper into the intricacies of model training and fine-tuning, focusing on the relational aspects of ontology schemas. Keywords: Knowledge Retrieval, SPARQL Queries, Prompt Engineering, Agents. Resumen: La irrupci´on de la Web Sem´antica ha precipitado una proliferaci´on de datos estructurados manifestados en forma de grafos de conocimiento, subrayando la imperativa necesidad de interfaces de lenguaje natural para mejorar la accesibilidad a estos repositorios de informaci´on. La capacidad de articular consultas en lenguaje natural y posteriormente recuperar datos a trav´es de consultas SPARQL asume una importancia primordial. En la presente investigaci´on, hemos analizado la eficacia de la t´ecnica de in-context learning usando una arquitectura basada en agentes para facilitar la construcci´on de consultas SPARQL. Contrariamente a las expectativas iniciales, se ha encontrado que la mejora del prompt de in-context learning a trav´es de mecanismos basados en agentes disminuye la eficacia de los Sistemas Basados en Modelos de Lenguaje (LLMS), al ser percibidos como ”ruido” extr´ınseco, mostrando as´ı las limitaciones inherentes de esta aproximaci´on. Los resultados resaltan la necesidad de profundizar en las t´ecnicas de entrenamiento y fine-tuning de modelos, centr´andose en los aspectos relacionales de los esquemas de ontolog´ıa. Palabras clave: Recuperaci´on de informaci´on, Consultas SPARQL, Ingenier´ıa de Prompts, Agentes. 1 Introduction The semantic web field has made remarkable progress thanks to the new computational paradigms and the increase in unstructured data over the last decade. This progress has led to a greater interest in representing this data within the linked data paradigm expressed as a knowledge graph. However, extracting information from this paradigm implies using the SPARQL query language. For building a SPARQL query, a human developer can formally follow at least the next three-stage methodology: a) identify the entities and relationships in the query, b) map the entities and relationships to the ontology, and c) SPARQL query generation. To be successful in this task implies a steep learning curve for those outside the computational domains and, therefore, a barrier to entry for non-experts in SPARQL Procesamiento del Lenguaje Natural, Revista nº 73, septiembre de 2024, pp. 271-281 recibido 04-04-2024 revisado 13-05-2024 aceptado 15-05-2024 ISSN 1135-5948 DOI 10.26342/2024-73-20 © 2024 Sociedad Española para el Procesamiento del Lenguaje Natural since both a deep understanding of the structure underlying the ontology used and fluency in the code’s syntax is necessary. To overcome this barrier, the research community has developed Question-Answering (QA) systems using Natural Language Processing (NLP), databases, information retrieval and machine learning techniques to generate SPARQL code comparable to the human judgement, (Liang et al., 2021). Despite considerable efforts to express human natural language queries into SPARQL language, the performance in this task remains low. This is primarily due to the lack of structural information related to the knowledge graph structure and the difficulty in isolating and understanding the entities and relationships involved in the human query. The advent of transformer architecture has allowed using the potential of encoder models in some of the stages of the problem, even some works have combined them with a Large Language Models (LLMs) (Yang et al., 2023; Taffa and Usbeck, 2023; Perevalov et al., 2022), especially the Code’s LLMs (Guo et al., 2024; Rozi`ere et al., 2024). However, to the best of our knowledge, no work has used LLMs for all the stages of a SPARQL query construction. This work introduces an LLM-agent-based architecture for translating queries formulated in natural language into SPARQL queries. The research questions raised are: (RQ1) To what extent are LLMs suitable for executing a three-stage methodology in the text-to-SPARQL translation task? (RQ2) How does the inclusion of ontology schema in the prompt influence the generation of SPARQL queries from natural language questions in terms of performance? (RQ3) Are LLMs capable of discerning the relevance of in-context information effectively amidst noisy data? (RQ3.1) In designing a system for converting natural language to SPARQL queries, is it more advantageous to gather a larger set of examples at the risk of including less relevant information, or should the focus be on ensuring the high relevance of the information even if it means having fewer examples? (RQ4) What is the impact of using agent-based methods on enhancing the system’s ability to translate natural language into SPARQL queries, compared to baseline approaches like few-shot learning? In this context, we have explored different prompts to build a text-to-SPARQL solution that leverages the cutting-edge capabilities of LLMs. Our system is designed to understand and translate complex natural language queries into SPARQL, facilitating seamless interaction with knowledge graphs. Furthermore, we address the challenge of finding adequate mappings between natural language questions and the employed semantic model in cases where human annotations are absent, through Synthetic Data Generation. This process enriches the ontology schema with necessary annotations, aiming to improve retrieval accuracy and query generation. We investigate a method for retrieving relevant entities of an ontology schema based on their natural language descriptions, studying its impact on the system’s accuracy in interpreting and generating responses to user queries. The paper is structured as follows: Section 2 discusses the related works. Section 3 describes the system architecture of our LLM-agent-based system. Section 4 presents the results of our evaluations. Finally, Section 5 draws conclusions and outlines potential future works. All code is available at our GitHub repository1. 2 Related Work The development of user-friendly query interfaces for individuals without computer expertise, enabling their interaction with databases, is not a novel concept. Early systems initially focused on addressing this challenge for SQL use cases, as SQL has been and remains the standard query language for relational databases. These first approaches designed rule-based, pattern-based, grammar-based or neural network-based systems to translate from natural language to SQL, (Popescu, Etzioni, and Kautz, 2003; Li, Yang, and Jagadish, 2007; Soru et al., 2018). With the influx of massive data production, the semantic web, linked data, and their representation by using knowledge 1https://github.com/CLSInfra-UNED/CLSAugur Adrián Ghajari, Salvador Ros, Álvaro Pérez 272 graphs emerged. This irruption implied using SPARQL language, so the interest in building an easy-to-use interface turned in this direction. We consider automatic SPARQL query building a more complex task than the SQL equivalent. This is due to the need to understand the complex aspects of syntax and semantics of the natural language question for an adequate mapping to the entities and relationship and their individual defined in the ontologies. The methodology to address this challenge involves constructing a pipeline of modules, each tailored to manage one of the three stages previously outlined in creating a SPARQL query. This process emphasizes the extraction of syntactic information concerning entities and relationships. This extracted data serves as input for simpler algorithms that construct the skeletal framework of the query. Additionally, these modules utilize lexicons and other resources to identify synonyms for nouns and verbs, further refining the query generation process (Liang et al., 2021). These stages have been covered with different approaches. A hybrid approach was presented in Querix, an interactive system where human intervention was needed to clarify ambiguities, (Kaufmann, Bernstein, and Zumstein, 2006). Later, the Panto system generated a lexicon based on the owl ontology definition and built a parse tree using the Spacy language model (Wang et al., 2007). Another later approach, CASIA@V2, used Markov logic networks (He et al., 2014) to build a question-answering system based on sentence detection and semantic mapping against the ontology. OniQ system, apart from using Spacy language models, included using Wordnet to look for synonyms for the words extracted from the user question (Dorob˘at¸ and Posea, 2020). The advent of LLMs, especially the code LLMs (Guo et al., 2024; Rozi`ere et al., 2024), has represented a significant leap forward in the field of NLP and have reformulated the SPARQL query construction. Following the same methodology, the different modules have been replaced with approaches based on language models. SGPT is a system that built a linguist context based on encoders to feed a GPT system fine-tuning in the ontology. Neither code nor evaluation is accessible,(Rony et al., 2022). In (Yang et al., 2023), proposed SPARQL systems based on LLMs for building SPARQL queries following the step of SGPT, using a knowledge graph to extract all the information. Recently, we studied the different prompt engineering techniques for use in generative models to build SPARQL queries, showing that the few-shot technique had the best performance. In contrast, any combination of techniques in the prompt construction produced a noise effect on the model, decreasing its performance. In this work, we analyse the result of redefining the three-stage methodology using an agentLLMs architecture and the influence of these agents in the prompt construction. 3 Methodology Building upon the foundation laid in the Related Work section, where we explored the advancements in few-shot learning and various text-to-SPARQL systems employing techniques such as BERT-like encoders and generative models, this section delves into the specific methodologies employed in our study. Recognizing the potential of LLMs in bridging these domains, our work aims to evaluate their efficacy in translating natural language queries into SPARQL queries, without resorting to the direct training of LLMs. At the core of our methodology is a structured system designed to translate user queries into SPARQL queries via a multistage process. Initially, an agent is tasked with extracting potential entities and relations from the user’s natural language question, translating this information into pairs of identifiers and definitions. These agents operate independently but collaborate with the Code-LLM to achieve the system’s objective. Each agent is equipped with an LLM to generate descriptions, distinguishing them from traditional modules. This translation paves the way for the following step, where a Retrieval-Augmented Generation (RAG) system utilizes the identifierdefinition pairs to obtain relevant ontology identifiers, thereby enriching the prompt for a more accurate and context-aware query generation. Next, a second RAG system, containing question-SPARQL query pairs, leverages few-shot learning (FSL) to further augment the prompt. Finally, the enriched prompt, combining ontology information, similar examples, and the user’s query, is processed by a generative model to produce the final SPARQL query, Figure 1. This structured approach not only seeks Querying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation 273 Figure 1: Agent-based Architecture. to leverage the capabilities of LLMs for question translation but also aims to improve the accuracy and relevance of the generated SPARQL queries through strategic retrieval and generation processes. While the methodologies described herein strive for ontology agnosticism, our empirical study is grounded in the application of the DBpedia 2016 knowledge database and ontology schema. This choice facilitates a replicability while providing a framework for evaluating the generalizability and efficacy of our approach across diverse ontology schemas. The following subsections provide a detailed account of each component of our methodology, setting the stage for a comprehensive evaluation of their effectiveness. 3.1 Synthetic Data Generation In the earlier discussion, we highlighted the challenge of translating natural language queries into SPARQL queries and identified synthetic data generation as a crucial step to enrich our ontology schema. Developed with an extensive array of entities and relations, the schema includes selective descriptive comments. Addressing the absence of these annotations, vital for deepening understanding and enabling refined information retrieval, has traditionally required manual intervention. To bridge this gap, we employ an LLM, specifically utilizing Mistral-7B-Instruct-v0.2(Jiang et al., 2023), to generate meaningful comments for these ontology identifiers, incorporating machinecreated descriptions for each missing annotation. These descriptions were integrated into the ontology as RDF triplets with the comment relation. Specifically, each entity and relation without a human-provided annotation received a generated description, formatted as a triplet (subject, predicate, object) where the predicate is a rdf:comment relation, and the object is the generated description. This ensures that all elements within the ontology are annotated, enhancing the retrieval capabilities of the RAG system. DBpedia, serving as the knowledge database foundation for our study, amalgamates Wikidata information into a cohesive ontology. While DBpedia’s schema delineates the primary structure of this ontology, it notably omits certain properties that, despite their absence, remain critical for constructing comprehensive SPARQL queries. In response to this challenge, we extracted properties pivotal to the queries under consideration, alongside an additional 10,000 randomly selected identifiers. This strategic selection process expanded our ontology schema to encompass a total of 13,416 unique identifiers, significantly enhancing the diversity and representativeness of our schema, while ensuring it better mirrors a real-world scenario. The enriched ontology schema, augmented with descriptive comments and a wider array of identifiers, serves as the foundation for our Ontology RAG system. This system aims to provide relevant information to the LLMs responsible for generating the final query, crucial for creating coherent and contextually relevant SPARQL queries. Therefore, the synthetic data generation phase not only addresses a significant gap within our ontology schema but also ensures a smooth transition to the RAG system, which will be detailed in the subsequent section. 3.2 Ontology Schema RAG Continuing from the enrichment of the ontology schema with synthetic data, we highlight the role of the Ontology RAG system in crafting SPARQL queries. The schema provides a detailed semantic framework that is essential for the system’s effectiveness in query formulation. To make this detailed information easily accessible, a RAG system has been deployed, to store and aptly retrieve data pertinent to user queries or identifier descriptions. Within the RAG system, each ontology entry is represented as a vector, including semantic embeddings from the comments of all classes and properties, as well as relevant Resource Description Framework (RDF) triplets. This configuration allows for entry selection via cosine similarity searches in a vector database, guided by the semantic embeddings of the given input text. Adrián Ghajari, Salvador Ros, Álvaro Pérez 274 This method ensures the identification of comments and triplets that best match the query’s intent. Upon identifying a relevant triplet, the system recovers all schema triplets with the same entity as their subject. It further explores the ontology graph to include entities directly related to these triplets, enriching the context provided to the generative model. This data is added tho the prompt in Turtle (TTL) format. The agents responsible for querying this RAG system, which are central to the next phase of our methodology, are tasked with mapping natural language questions to potential entities, relations, and descriptions within the schema. These mappings will subsequently be used to query the Ontology Schema RAG. Details of this process will be provided in the following section. 3.3 Agents From our exploration of the enriched ontology schema, we turn our focus to the agents that extract and define relevant entities and relations from natural language queries to leverage this schema. We examine two approaches: PoS-based extraction and LLMbased extraction, both crucial for converting user queries into structured data that align with the RDF environment, utilizing the detailed ontology schema. 3.3.1 PoS-based Extraction For the first technique, we employ Stanford’s Stanza model (Qi et al., 2020) to extract relevant entities and relations from natural language queries. This model is a robust NLP toolkit that facilitates the extraction process by accurately identifying entities and their relations within the input text. Through syntactic and semantic analysis, Stanza identifies nouns, verbs, and interrogative pronouns and adverbs Part of speech (PoS) tags, thereby capturing the structural and semantic elements of the query. Once the relevant data is extracted, it is passed to an LLM for further processing. Specifically, the LLM generates concise definitions for each element extracted from the query. This process is aimed at retrieving relevant information from the RAG, as previously explained. 3.3.2 LLM-based Extraction In the second technique, we solely rely on an LLM to extract these elements and create definitions for all relevant entities and relations identified in the natural language queries. By leveraging the contextual understanding and knowledge representation capabilities of LLMs, this approach ensures that the extracted data is transformed into a structured format suitable for subsequent query processing within the RDF environment, bypassing the need for a separate tool such as Stanza for initial data extraction. This streamlined process enhances efficiency and coherence in the generation of definitions, facilitating seamless retrieval of relevant information from the RAG. The LLM directly processes the input queries, discerning the entities and relations based on the contextual information provided. Subsequently, it generates succinct definitions for each entity and relation identified in the query. This end-to-end approach not only streamlines the extraction process but also ensures consistency and coherence in the definitions generated, as they are crafted within the same model that comprehends the query. This method offers a seamless and efficient solution for extracting and defining entities and relations from natural language queries in the context of Natural Language Interfaces to Database (NLIDB) systems. Again, this process is conducted with the objective of retrieving relevant information from the RAG, as previously elucidated. 3.4 Few-shot Learning In the progression of integrating the ontology schema, our approach incorporates the utilization of the LC-QuAD dataset (Trivedi et al., 2017), a Linked Data Question Answering Dataset designed to replicate the human expertise in formulating SPARQL queries through example-based learning. This dataset, comprising over 5,000 pairings of questions and SPARQL queries, spans a wide array of question complexities, predominantly centered around DBpedia, making it exceptionally suitable for challenges in semantic parsing and question answering. For the enhancement of the RAG system, we have chosen a subset consisting of 4,000 pairings. This subset is utilized such that each pairing’s semantic embedding of the question serves as a key, with the pairing itself as the stored value. When a user submits a query, the query is projected into an embedding space within Querying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation 275 the RAG system. This embedding is then compared to the stored embeddings of the questions in the LC-QuAD dataset using cosine similarity. The system retrieves the five most similar questions based on this comparison. By retrieving the pairs of questions and SPARQL queries that are most similar to the user’s query, the system can leverage these examples to construct a more accurate and relevant SPARQL query for the user’s input. This method ensures that the model is exposed to pertinent examples, effectively utilizing real-life examples to refine its outputs. The combination of this example-based learning with the information provided by the agents enhances the overall accuracy and relevance of the generated SPARQL queries. 4 Evaluation In the present section, we move forward from the groundwork established in the Methodology, where we described our approach for translating natural language queries into SPARQL queries using an LLM-agent-based architecture. Here, we aim to demonstrate the practical outcomes of our experimental investigation. In our experimental evaluation, we utilized three models with distinct training characteristics and sizes: DeepSeek-Code 6.7B (Guo et al., 2024), CodeLlama-13B (Rozi`ere et al., 2024), and GPT 3.5 TURBO. This diversity allowed for a systematic analysis of the models’ ability to translate natural language into SPARQL queries. Our evaluation was twofold, focusing initially on the system capability to extract pertinent information from the schema, a process we will refer to as ”method evaluation”. This was treated as a multilabel classification problem, comparing the URIs identified by each system against the gold standard. To further clarify, we extract the unique URIs contained in each generated prompt, where all the information added by the different subsystems converge. We do this process for every method, capturing the elements that each evaluated method deems relevant. Similarly, we also extract the unique URIs from the SPARQL query in the gold standard to identify the elements that are present, comparing them with the ones the system has been able to extract for that specific user query from the test set. This comparison allows us to calculate precision, recall, and F1 scores, providing insights into the system’s ability to identify relevant instances and its overall performance per method. The Jaccard index, a measure of set similarity, calculated as the intersection over the union of the predicted and actual relevant URIs, served to highlight the models’ efficiency in isolating relevant information. Four experimental approaches were conducted for each model: Naive, involving a simplified architecture where user questions were directly processed by the ontology RAG system to retrieve relevant entities and relations, bypassing the agents; PoS and Gen, both agent-based approaches for enhanced data retrieval accuracy; and the FSL method, serving as our baseline without additional ontology schema information. In the second phase of our evaluation, termed ”execution evaluation,” we assessed the system’s capability to generate effective and structurally correct SPARQL queries. This evaluation focused on the system’s overall performance and its ability to handle different types of queries, including ASK (83 samples), COUNT (123 samples), and SELECT (794 samples). This analysis aids in discerning the system’s proficiency and adaptability across varied query scenarios. For these evaluations, we used the test subset of the LC-QuAD dataset, consisting of 1,000 question-SPARQL pairings. This dataset allowed us to test the system’s capabilities in a controlled yet varied set of conditions. The effectiveness and accuracy of the generated queries were measured in percentage values, providing a straightforward method for comparing the performance of different models and approaches. 4.1 Experiment Results Building on the setup described in the section above, where we explored the methodologies for translating natural language queries into SPARQL queries using the DeepSeek-6.7B, CodeLlama-13B, and ChatGPT 3.5 models, we present the findings of our experiments. As delineated in Table 4, the FSL approach significantly outshined other methods (Naive, PoS, and Gen) in terms of F1 score, Precision, and Jaccard index. The FSL method achieved a notable F1 score peak of 20.87%, a marked increase from the Naive method’s lowest score of 12.98%. Precision and Jaccard metrics exhibited parallel trends, with the FSL method reaching the Adrián Ghajari, Salvador Ros, Álvaro Pérez 276 Method Performance Metrics (%) Ex Eval (%) Model/Method F1 Recall Precision Jaccard Acc Valid DeepSeek-6.7B/Naive 12.98 66.48 7.27 7.12 29.9 80.7 DeepSeek-6.7B/PoS 13.65 67.16 7.67 7.51 34.0 97.5 DeepSeek-6.7B/Gen 14.79 66.95 8.41 8.22 35.2 97.5 DeepSeek-6.7B/FSL 20.87 65.05 12.70 12.25 37.4 98.4 CodeLlama-13B/Naive 12.98 66.48 7.27 7.12 33.2 95.8 CodeLlama-13B/PoS 13.71 66.39 7.72 7.55 37.8 98.8 CodeLlama-13B/Gen 14.18 67.03 8.02 7.85 37.8 98.7 CodeLlama-13B/FSL 20.87 65.05 12.70 12.25 41.0 99.5 ChatGPT 3.5/Naive 12.98 66.48 7.27 7.12 34.3 75.5 ChatGPT 3.5/PoS 13.76 67.03 7.75 7.58 39.4 99.0 ChatGPT 3.5/Gen 14.45 67.57 8.19 8.01 38.6 99.6 ChatGPT 3.5/FSL 20.87 65.05 12.70 12.25 47.7 99.5 Table 1: Method Performance and Execution Evaluation Across Models. Gen Agent. (%) PoS Agent. (%) Model ASK COUNT SELECT ASK COUNT SELECT ChatGPT 3.5 50.60 34.15 38.04 51.81 34.15 38.92 CodeLlama-13B 45.78 28.46 38.41 45.78 30.89 38.04 DeepSeek-6.7B 40.96 30.89 35.26 44.58 28.46 33.75 Table 2: Accuracy Evaluation Across Models and Query Types. highest Precision of 12.70% and Jaccard index of 12.25%, in stark contrast to the Naive method’s 7.27% and 7.12%, respectively. The accuracy and validity metrics were also led by the FSL method, with ChatGPT 3.5 achieving the highest accuracy of 47.7% and a validity of 99.5%. Interestingly, ChatGPT 3.5 under the PoS method outperformed the Gen method in accuracy (39.4% versus 38.6%), even though the latter had a slightly higher Jaccard index and recall. This suggests that larger models like ChatGPT might be better equipped to navigate through noisy data, leveraging their extensive context window to prioritize relevance in information retrieval. Notably, the FSL method achieved the best accuracy for all models while having the lowest recall of all methods, but it boasted the highest Jaccard index, indicating that it managed to retrieve the most relevant information. This underscores the importance of the quality of information over sheer quantity, particularly in the context of generating structurally correct and effective SPARQL queries. CodeLlama-13B exhibited identical accuracy for both the Gen and PoS methods (37.8%), despite the Gen method having a marginally higher recall and Jaccard index. This scenario mirrors the situation in the ChatGPT model, albeit with a smaller differential, hinting at a nuanced interplay between model size, recall, and the ability to handle noise. For DeepSeek-6.7B, the PoS method, despite having a higher recall, achieved a lower accuracy (34.0%) compared to the Gen method (35.2%), indicating that the relevance of inputs might have a more pronounced effect on smaller models. Diving into the query-type-specific analysis in Table 4, ChatGPT 3.5, under the PoS method, achieved the highest accuracy in ASK queries at 51.81%, and both Gen and PoS methods tied at 34.15% for COUNT queries. For SELECT queries, the highest accuracy was recorded at 38.92% by ChatGPT 3.5 using the PoS method. Conversely, DeepSeek-6.7B exhibited the lowest performance in ASK and SELECT queries, with the minimum accuracy of 40.96% noted for ASK queries under the Gen method. These observations underscore the significant variability in the performance of difQuerying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation 277 ferent models and methods, with the FSL method consistently showing superior performance in generating SPARQL queries. The detailed analysis emphasizes the need for a nuanced selection of model and method in the natural language to SPARQL query translation tasks, considering not only the model’s size and methodological approach but also its capacity to process and prioritize relevant information amid potential data noise. 4.2 Discussion The experimental results shed light on several key aspects of model performance in translating natural language to SPARQL queries. The interplay between the Jaccard index, recall, and model performance (accuracy) underscores a pivotal aspect of generative LLMs in handling noise during inference. The data shows that there is a notable relationship between the Jaccard index and the accuracy of generative LLMs, with the former potentially having a greater impact than recall. Our analysis suggests that a higher Jaccard index, indicative of providing more relevant identifiers from the ontology schema, enhances model performance more effectively than simply increasing the number of correct identifiers, especially if it also introduces more incorrect ones. This observation leads us to argue that models prefer contexts with lower noise levels, emphasizing the importance of precision in the input data. In addition, our findings indicate that all LLM agentbased approaches (PoS and Gen) consistently achieved a higher Jaccard index in all cases, and higher recall in all but one case, compared to the Naive approach. This suggests that the agent-based methods are more effective in filtering relevant information from the schema, demonstrating a nuanced comprehension of the schema’s components and their interrelations. Furthermore, we discuss the role of model size in the resilience to noisy inputs. ChatGPT shows an interesting trend where a method with more noise still outperforms one with a higher Jaccard index. This phenomenon might indicate that while larger models can handle more noise, they also extract value from increased contextual information, balancing between recall and precision. Conversely, DeepSeek, being the smallest model in our evaluation, seems most sensitive to the Jaccard index, indicating a preference for cleaner, more relevant context. CodeLlama-13B falls in between, exhibiting resistance to noise, and might be partially attributed to its ability to navigate the trade-off between recall and precision effectively. A challenge highlighted by our study is the difficulty LLMs face in applying ontology schemas to generate SPARQL queries. The models often struggle beyond copying URIs, suggesting a gap in training where schemas and SPARQL queries are not presented together frequently or comprehensively. This gap likely hinders the model’s ability to learn and apply the relational structure inherent in ontology schemas, especially as these schemas become extensive and complex. The ”Reversal Curse” phenomenon (Golovneva et al., 2024) further complicates this issue. This term refers to a model’s difficulty in generalizing from one context to another; specifically, when trained on data where A has feature B, the model does not easily generalize that B can be a feature of A, despite being trained on many pairs of such facts. Applying this to our context, LLMs trained on data where ontology classes, relations, and properties are distinct may not effectively generalize that these elements are interconnected in SPARQL queries, and cannot be viewed in isolation. Lastly, we consider the challenges posed by in-context learning. Although LLMs can adapt based on the context provided during inference, the intricate and dynamic nature of ontology schemas demands a level of understanding that goes beyond simple pattern recognition. We contend that LLMs need to not only identify relevant schema components but also comprehend their interconnections and apply them correctly within queries, a task that necessitates a deep, contextual understanding, which is challenging to achieve through in-context learning alone. Our discussion reveals the intricate balance between providing models with precise, relevant context and the breadth of information necessary for effective query generation. The complexity of ontology schemas, combined with the inherent limitations of LLMs in schema interpretation and in-context learning, poses substantial challenges that future research must address to enhance the accuracy and effectiveness of natural language to SPARQL translation. Adrián Ghajari, Salvador Ros, Álvaro Pérez 278 5 Conclusions and Future Work In this paper, we explored the efficacy of LLMs in translating natural language queries into SPARQL queries through an agentbased architecture. We found that LLMs, including those designed for code generation, still face challenges in accurately translating natural language into SPARQL queries that effectively capture the intended information from the ontology database. This addresses RQ1, suggesting that while LLMs are proficient in many areas of language understanding and generation, the specialized nature of SPARQL query construction, particularly in reflecting complex ontology structures, remains a significant hurdle. The inclusion of ontology schema in the prompt was often perceived by the model as added noise, rather than as helpful context. This observation hints at an underlying issue where LLMs struggle to form meaningful connections between the ontology schema and the SPARQL queries. The schema’s complexity and the model’s training on potentially disparate datasets may contribute to this disconnect, indicating a gap in the model’s ability to integrate and apply interconnected schema knowledge in query generation (RQ2). Our experiments underscored the challenge LLMs face in integrating ontology schema into SPARQL queries, with the Reversal Curse phenomenon indicating a critical gap in model generalization capabilities. This gap, between the relational knowledge of schema elements and their application in query generation, poses a notable obstacle in the path of achieving more accurate and contextually relevant query translations. In response to RQ3, our evaluation suggests that larger models like ChatGPT have a better capacity to manage noisy data but show improved performance when the input is more precisely tailored with relevant identifiers. This underscores the importance of precision and the quality of information over its quantity, underlining that a well-curated input plays a crucial role in enhancing the model’s output accuracy. The results of our study underscore the value of providing more relevant and precise information rather than a larger quantity of potentially noisy data. The model’s performance is negatively impacted by irrelevant data, reinforcing the idea that cleaner, more relevant inputs lead to more accurate and useful SPARQL query outputs. Moreover, the model’s size influences its ability to process and filter out noise, with larger models showing a greater tolerance for irrelevant data but still benefiting from a more relevant information set. This conclusion directly responds to RQ3.1, illustrating the balance between data relevance and quantity in optimizing model output. Finally, RQ4 is answered through the observed efficacy of agent-based methods, which have consistently outperformed the Naive approach in our experiments. This improvement indicates that agent-based systems, with their ability to parse, interpret, and select relevant information before query generation, provide an advantage in translating natural language into SPARQL queries. Looking forward, the limitations identified in in-context learning suggest that a shift towards fine-tuning approaches may yield better outcomes. Future work should consider Causal Language Modeling (CLM) finetuning, where models are trained explicitly with schema, question, and SPARQL query triples, ensuring both schema and code are normalized. This approach would necessitate that the model not only learns the syntax of SPARQL queries but also deeply understands the relational dynamics present in generic schema/code constructs. By doing so, the model could develop a more nuanced and robust framework for translating natural language into SPARQL queries, potentially overcoming the challenges highlighted by the Reversal Curse and enhancing the model’s ability to interpret and apply complex ontology schemas accurately. In conclusion, while our current methodologies provide a foundational understanding and some level of success in translating natural language to SPARQL queries using LLMs, there remains a substantial scope for improvement. Future research will need to delve deeper into the intricacies of model training and fine-tuning, focusing on the relational aspects of ontology schemas to truly harness the potential of LLMs in this domain. Acknowledgements This research has been carried out in the Grant CLSINFRA reference 101004984 framework funded by EXCELLENT SCIENCE. Querying the Depths: Unveiling the Strengths and Struggles of Large Language Models in SPARQL Generation 279