Using AI to identify papers using ESO data

AI Integration into ESO Library Fulltext Search (FUSE) & Telescope Bibliography (telbib) Enhancing ESO’s Telescope Bibliography through Machine Learning Mahmut Eren, Silvia Meakins, Uta Grothkopf, Leslie Kiefer ESO Library, Documentation, and Information Services department Based on the project completed by Claudio Urbina as a master thesis supervised by Amelia Bayo and Mauricio Cerda LISA 10 | 4 November 2025 AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference 1 Data Classification: PUBLIC 2 Agenda •Introduction Telbib & FUSE •Motivation for AI Integration? •How AI Works in FUSE •Model Performance •Future Work •Summary AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 3 ESO TELESCOPE BIBLIOGRAPHY telbib: db of refereed papers that use ESO data •built and curated by ESO Librarians •interlinks resources: ESO data papers and the observations in the Science Archive, and to/from other resources •tool to evaluate ESO’s performance and impact through bibliometrics AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Coding / scripting Data ingestion and curation Strategic development telbib.eso.org ESO Science Archive ALMA Science Archive 4 Telbib Paper Identification Process AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC telbib FUSE 5 Telbib Paper Identification Process Manual Classification Curators decide to include/exclude the paper . Tagging Curators tags the paper. Weekly Search Search papers published in journals that are routinely searched. Fulltext Search Grab PDF for the paper, convert it to txt, search for keywords telbib FUSE AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 6 FULLTEXT SEARCH TOOL - Keywords AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 7 FULLTEXT SEARCH TOOL - Keywords AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Includes telescopes, instruments, surveys Case-sensitive with regex support Automatically identify candidate ESO papers Curators maintain and expand 8 FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 9 FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Weekly ADS search 16 FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC ESO instruments, survey names, etc., Direct access to paper content Found keywords Program IDs/Data DOIs 17 ESO instruments, survey names, etc., Direct access to paper content Found keywords Program IDs/Data DOIs FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Papers that ‣partly or exclusively use ESO data (proprietary or archival) included Papers that ‣quote results from literature ‣mention ongoing projects ‣suggest future observations ‣describe instrumentation / software ‣develop models or run simulations merely as examples or for training (e.g., Machine Learning) ‣use images only as visual reference excluded http://www.eso.org/sci/libraries/telbib_methodology.html 18 Motivation for AI Integration Enabling more efficient workflow as number of publications and ESO facilities are increasing AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Need for consistent classification of ESO data papers Data Classification: PUBLIC •Claudio Urbina (thesis) •Amelia Bayo, ESO •Mauricio Cerda, Univ. Chile •ESO curators Simplified onboarding for new curators and contributors 19 AI Integration - Roadmap AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Telbib data extraction Preparing the training dataset from Telbib records Experimentation Evaluating different Model architectures Model scope definition Defining separate models for Purpose and Usage classification AI service development Implementing an API-based service for model inference. Training of two models Developing and optimizing PurposeModel and UsageModel Integration Into FUSE Connecting the AI service with FUSE for automatic paper assessment. 20 AI Models Purpose Model Usage Model •Trained with paper abstracts •Classify scientific vs. non-scientific •Trained with paper sentences with ESO keywords •Detect ESO observational data usage AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 21 Telbib Paper Identification Process Weekly Search Search papers published in journals that are routinely searched. Fulltext Search Grab PDF for the paper, convert it to txt, search for keywords FUSE AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Manual Classification Curators decide to include/exclude the paper . Tagging Curators tags the paper. telbib 22 Telbib Paper Identification Process Weekly Search Search papers published in journals that are routinely searched. Fulltext Search Grab PDF for the paper, convert it to txt, search for keywords FUSE AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Manual Classification Curators decide to include/exclude the paper . Tagging Curators tags the paper. telbib AI Classification Predict if scientific discovery paper and ESO data usage 23 FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 24 FULLTEXT SEARCH TOOL AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Purpose model result Usage model result 25 FUSE + AI Interface: Model Threshold Values AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Purpose model result Usage model result Probability (p) Threshold (t) 0.99 0.93 32 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 33 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 34 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 35 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 36 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 37 FUSE + AI Interface AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 38 FUSE + AI Interface: Usage Model Result AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC 39 How Well Does the AI Work? Curator Decided Positive Curator Decided Negative AI Predicted Positive 261 (correct) 14 (false alarms) AI Predicted Negative 20 (missed) 174 (correct) “High and consistent performance across all key indicators..” AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC So far, we assessed 469 papers using AI. 40 Future Work AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC Optimize model parameters for accuracy Expand training data to include more recent publications Integrate user feedback to refine classification over time related publications 41 Summary AI Integration into ESO Library Fulltext Search (FUSE) &Telescope Bibliography (telbib), 4 Nov 2025, LISA 10 Conference Data Classification: PUBLIC The models enable more consistent classification of ESO-related publications Curators appreciate the additional tool Early results promising, continuous improvements ongoing Thank You [email protected]