Full text
FAIR, fast, and frictionless –computational workflows with SciWIn Jens Krumsieck1, Antonia Leidel2, Xaver Stiensmeier3, Harald von Waldow1 1: Johann Heinrich von Thünen-Institute, Braunschweig, Germany 2: Leibniz Institute of Plant Genetics and Crop Plant Research, Gatersleben, Germany 3: Bielefeld University, Bielefeld, Germany Scientific Workflow Infrastructure CSV md { } json >s4n create –c python:3.13 --enable-network python analysis.py --data raw.csv --strict true Metadata Input Parameters Output Parameters Runtime Environment Command tracks file changes git Repository CWL Added in commit CWL CommandLineTool >s4n connect my_workflow --from raw_data --to analysis/data Metadata Input Parameters Output Parameters Workflow Steps >s4n connect my_workflow --from analysis/data --to result_data CWL CWL Workflow >s4n visualize workflows/my_workflow/my_workflow.cwl cwlVersion: v1.2 class:CommandLineTool requirements: -class:DockerRequirement dockerPull: osgeo/gdal:ubuntufull-3.6.3 inputs: -id: filename type:string default: districts.geojson inputBinding: position: 0 outputs: -id: districts type:File outputBinding: glob: $(inputs.filename) baseCommand: ogr2ogr cwlVersion: v1.2 class:Workflow inputs: -id: token type:string outputs: -id: raw_data type:File outputSource: collect/raw_data steps: -id: collect in: -id: token source: token run: ../collect/collect.cwl out: - raw_data CommandLineTool Workflow Metadata Input Parameters Output Parameters Runtime Environment Command Metadata Input Parameters Output Parameters Workflow Steps CWL Building Blocks Common Workflow Language (CWL) Automated creation of CWL CommandLineTools Creation of complex CWL Workflows by connecting Tools and Workflows THE PROBLEM •Reproducible computational research is challenging •Scientific computational workflows as a solution BUT: •More than 300 Workflow Management Systems to choose 🤯 •Researchers spend hours to write boilerplate code for Workflow definitions 😵 •Open standard for workflow definitions •Metadata describing what not how to run •Based on YAML (machine & human readable) •Portable >s4n execute remote start workflows/my_workflow/my_workflow.cwl inputs.yml KEY FEATURES •Command-line application (s4n), written in Rust •Automated generation of CWL Files •Workflow Visualisation (publication ready figures) •Version-controlled workflows using git •Easy reuse existing Workflows (via git submodules) •Local execution of CWL Files •Remote execution using Reana, in cooperation with de.NBI •Export reana runs as provenance run crate •Compatible with DataPLANT‘s ARC format COMING SOON •WorkflowHub.eu integration •(Semi-)automated Dockerfile creation •Graphical User Interface >s4n execute local workflows/my_workflow/my_workflow.cwl inputs.yml reana.bi.denbi.de REST API Workflow Execution local or at scale github.com/fairagro/m4.4_sciwin_client Iillustration by https://www.manypixels.co