DataMap: A Scalable MicroservicesBased Platform for Democratising and Analysing Massive Biological Knowledge Graphs Sadegh Abadijou Senior Software Engineer UK DRI Core Informatics
UK Dementia Research Institute Dementia is a neurodegenerative condition in which brain neurons progressively die. •Memory loss •Difficulty concentrating •Problem with language and communication By the end of this presentation, nearly 300 people worldwide will be diagnosed with dementia.
UK Dementia Research Institute The UK’s leading research institute transforming the lives of people affected by neurodegenerative conditions •Discover what causes dementia •Develop new tools, tests, and treatments •Deliver solutions to maintain brain health in an aging society 10 centers across the UK Collectively UK DRI scientists generate terabytes of data monthly. Our mission is to
The problem •Insights remain trapped in individual labs So no institute-wide integration •No unified platform to analyse or democratise this knowledge •No platform for integrating knowledge •Legacy tools fail or are slow •Most tools are inaccessible to non-technical researchers ~100s of TB/month from genomics, imaging, transcriptomics, proteomics, behaviour
DataMap: supports seamless knowledge integration for biological datasets •Data integration •Facilitating the harmonisation and utilisation of heterogeneous datasets across the UK DRI •Data democratisation •A secure tool for making UK DRI data accessible and interpretable to the widest range of stakeholders through a web application. •Investigation tools •Provide researchers with tools that enable interaction with a complex biological knowledge graph and exploration of its various layers DataMap is an interactive, graph - based platform designed to explore and analyse complex biological systems.
DataMap: From ingestion to insight •Interoperable by design and modular architecture •Cross-omics integration for multi-layered analysis across genes, proteins, phenotypes, and traits •Interactive graph exploration •FAIR-Compliant & open •User-centric Design: Define and refine user requirements, iteratively adapting the software architecture and features based on continuous researcher feedback.
Technical challenges •Graph performance at scale •Slow traversals in large biological networks •Heavy computational workloads •Long-running tasks like statistical evaluation across a co-expression network on the fly •Frontend visualisation bottlenecks because of physical complexity •Scalable deployment •Usability for non-technical users
DataMap’s microservice-oriented solutions •Modular microservices with flexible graph core •Each service focuses on a specific task, built around a Neo4j-based knowledge graph with dynamic schema handling via metaprogramming. •High-performance, In-memory acceleration •Redis Stack enables rapid lookups, cached traversals, and precomputed layouts •Asynchronous task management via message passing •Redis Streams acts as a message bus for decoupled microservices •Secure, Scalable, and User-Centric by design •Keycloak for federated identity and RBAC, intuitive TypeScript+React+D3 Force Package interface, and containerised deployment.
Architecture: Microservices •Handling computationally intensive, parallel workloads •Scalable data ingestion and processing •Flexibility to use specialised tools/frameworks •Modularisation of complex pipelines •Easier collaboration across teams
Thank you! Sadegh Abadijou Senior Software Engineer
[email protected] ukdri.ac.uk Let’s stay in touch on LinkedIn With sincere appreciation to our team: •Caleb Webber •Director of Informatics •Amonida Zadissa •Associate Director of Informatics •Dammy Shittu •Theme data wrangler New Talent, New Possibilities •Nikolai Hecker •Aaron Pangilinan •Li Ling Lee
