Automated Data Integration from Heterogeneous Sources using LinkAhead
Spreckelsen, Florian; Schlemmer, Alexander; Fitschen, Timm Christian; tom Wörden, Henrik
- Publisher
- Zenodo
- Language
- en
Abstract
Many scientific projects rely on a multitude of different software systems for data storage and data exchange. Keeping data findable and accessible can be challenging, especially if data has to be shared between different sites, working groups and institutes. The open source software LinkAhead provides a powerful framework for managing complex data integrated from heterogeneous data sources.LinkAhead is built in a way that data models can be extended and adapted to future requirements by the researchers at any time. Its extendable crawler framework can be adapted to automatically import data and meta data from different repositories and ELN systems, like elabFTW or PANGAEA. LinkAhead also provides a graphical web-interface, as well as an API for automated queries.
Full text
Automated Data Integration from Heterogeneous Sources using LinkAhead FAIR in Action 2025-10-02 Florian Spreckelsen and Alexander Schlemmer
What is LinkAhead? A 290 A 121 Exp 002 Lab data (ELNs, ...) Processing, Data Analysis Numerical Simulations Experimental Results Prüfzertifikat gemessene Werte: Ist: 358.12 Norm-Abweichung: 21.83 nach DIN ISO PRÜFSTELLE Publications Ideas, Protocols, Internal Documents, SOP 1 2 3 4 LinkAhead Thinking Data Management Ahead
Agile, semantic data management ●LinkAhead initially developed at MPI-DS (Göttingen) ●Released as open-source (AGPLv3) in 2018 (gitlab.com/linkahead) ●Commercial support by IndiScale GmbH LinkAhead Thinking Data Management Ahead
LinkAhead and ELNs ●Make data more findable ●Link data (e.g. ELN with simulations) ●Support semantic queries, e.g.: FIND Dataset which is referenced by Simulation with recording = 15 ●WIP: LLM-based query interface ”Find all data sets with links from simulation with recording equal to 15”
The LinkAhead Crawler
ELNFile : type : ELNFile match: ^presentation-dataset\.eln$ subtree : AuthorDataset : type : ROCrateEntity match_entity_type : Person match_properties : name: (?P<name>.*)$ "@id" : (?P<url>.*)$ records : Author : url : $url name: $name
Data Model
Live Demo ●Synchronize data: elabFTW → LinkAhead (Crawler + elabFTW-API) ●Synchronize data: Filesystem → LinkAhead (Experiment + Analysis) ●Connect experiment from elabFTW with analysis from file system
