Full text
www.ebi.ac.uk Dalya Al-Shahrabi, Prasad Basutkar, Renato Caminha Juaçaba Neto, Vijay Subramoniam, Rose Neis, Iva Tutis, Henning Hermjakob, Matt Pearce EMBL’s European Bioinformatics Institute (EMBL-EBI) Wellcome Genome Campus, Hinxton, Cambridgeshire. CB10 1SD. UK. T +44(0)1223 494 444 EBI Search (www.ebi.ac.uk/ebisearch/) is a unified metadata search engine that indexes biological data from 170+ internal and external sources, providing users with data discovery tools such as free-text search, multi-level faceting, cross-referencing capabilities, and bulk querying across the indexed metadata. Built on Lucene Core [1], it offers both web interface and RESTful APIs for programmatic access [2]. EBI Search: Engineering and Sustaining Metadata Infrastructure for Life Sciences 6.8 billion total entries 170+ data sources (domains) 2TB indexed data 26,000 unique users per month 387,000 unique API users per month References 1. Apache Lucene Core [Internet]. Lucene.apache.org. 2022 [cited 30 August 2022]. Available from: https://lucene.apache.org/core/ 2. Matthew Pearce, Prasad Basutkar, Renato Caminha Juaçaba Neto, Vijay Venkatesh Subramoniam, Kelsey Neis, Iva Tutis, Henning Hermjakob, EBI Search: providing discovery tools for biological metadata in 2025, Nucleic Acids Research, 2025;, gkaf359, https://doi.org/10.1093/nar/gkaf359 Acknowledgements EMBL-EBI is indebted to its funders, including the EMBL member states and the European Commission through the H2020 Programme under EOSC-Life [824087], BY-COVID [101046203], and EarlyCause [84815]. Nightly Indexing Pipeline The nightly indexing pipeline processes updated data from internal and external sources using parallelized execution managed by Slurm. Indexing Pipeline Slurm job scheduler manages concurrent execution across all domain indexing jobs 170+ Data Sources EMBL-EBI resources & external datasets XML, JSON, flat file, bioinformatics formats, … Data Update check mtime, 304 http status, fetch over HTTP, FTP, filesystem Data Splitting regex-based file splitting, memory chunking for parallel processing Lucene Indexing apply domain-specific analysers, generate Lucene documents and index files Verification validate cross references, check data consistency Deployment copy indexes to production, update "latest" pointers EBI Search API Website RESTful API External sites Engineering Decisions Increasing data volumes and user demands create competing technology requirements. The engineering decisions we make in response shape the EBI Search infrastructure. Reusable Portal Components Reusable templates and components streamline EBI Search portal creation, avoiding duplicated effort while preserving performance and feel. Index Partitioning Work around Lucene's 2.1 billion document limit through hierarchical domain splitting with transparent query distribution. Query rewriting Avoid storage duplication by rewriting text searches to target multiple fields at query time rather than creating combined fields. API Optimisations Separate filter queries from relevance scoring for better search performance, and enable bulk streaming for results exceeding 1 million entries. Future Directions Scaling infrastructure To better support the indexing of metadata for tens of billions of records, we are migrating to Apache Solr. This will enhance both scalability and efficiency, and enable more seamless search experiences across multiple datasets. DocBot Developing DocBot, a chatbot that provides user-friendly access to EBI’s extensive documentation resources. This pilot project develops internal expertise in LLM deployment, laying the foundation for a comprehensive SearchBot integrated with EBI Search and spanning most EBI content. Beyond the Code EBI Search is run and maintained by a small team of software engineers. Daily decisions, provider relationships, and systematic troubleshooting are critical for long-term sustainability. Daily monitoring Email alerts, failure investigation, immediate issues Domain management Manual domain retirement and rollback decisions Provider relationships Negotiating formats, ongoing support, proactive communication when APIs fail Technology evolution Weighing infrastructure improvements against risk factors for continued operation
