FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs

Experiments and Results [email protected] Pilot experiments: using ChatGPT to generate SPARQL query for handcrafted questions with one-shot learning Natural language questions LLMs SPARQL queries Semantic inaccuracies Problem Fail to link the correct properties and entities in ORKG What is the maximum sample size? Contribution Evaluation Metric P34 P2006 P7046 Structural inconsistencies Problem Make errors in query structure, such as missing or abundant links (triples) What are the metrics used by paper "Using NMF-based text summarization to improve supervised and unsupervised classification? orkgp:P15687 rdfs:label Sample size (n) orkgp:P7101 rdfs:label has elements Methodology Task definition Motivation FIRESPARQL: A LLM-BASED FRAMEWORK FOR SPARQL QUERY GENERATION OVER SCHOLARLY KNOWLEDGE GRAPHS Xueli Pan, Victor de Boer, Jacco van Ossenbruggen Datasets SciQA Benchmark, 100 handcrafted questions (HQs), 2465 automatically generated questions (AQs). Implementation LoRa fine-tuning on Llama 3.2-3B Instruct and Llama 3-8B Instruct, using 1,795 NLQ-SPARQL pairs from the SciQA training set. The models were trained under various epoch configurations (3, 5, 7, 10, 15, and 20) DeepSeek-R1-Distill-Llama-70B for the RAG Qlever for SPARQL execution a single NVIDIA H100GPU Findings Fine-tuning significantly improves accuracy RAG alone does not guarantee improvement — context quality matters SPARQL Corrector module improved syntactic validity and execution success. HQs remain challenging, AQs are easier because of repetitive templates and consistent structure. Bridging the gap between natural language and structured scholarly data Addressing the limitations of LLMs in scholarly knowledge graphs (SKGs) parsing Improving accuracy of generating SPARQL from natural language questions (NLQ) Metric Contribution [1] Auer, Sören, et al. "The sciqa scientific question answering benchmark for scholarly knowledge." Scientific Reports 13.1 (2023): 7240.