scieee AI-readable full text Open interactive document viewer

PAIDIVERPY: An Open-Source Python package designed to create pipelines for preprocessing image data for biodiversity analysis.

Sauze, Colin; Durden, Jennifer; Ferreira, Tobias; Masoudi, Mojtaba; Van Audenhaege, Loïc; Orenstein, Eric

Publisher
Zenodo
DOI
10.5281/zenodo.17967415
Repository
zenodo.org
Language
en

Abstract

Biologists and ecologists are increasingly using imaging systems to study the natural world. Getting meaningful biodiversity metrics from visual data requires reliable, reproducible image preprocessing. However, existing tools for image preprocessing are fragmented, application-specific, not well documented, and not designed for the needs of ecologists. This lack of standardization makes reproducibility difficult and slows down scientific progress. To address these challenges, we present PAIDIVERPY—an open-source modular Python package designed to simplify, standardize, and document image preprocessing workflows for biodiversity analysis. PAIDIVERPY is a flexible framework for building reproducible image preprocessing pipelines, complete with detailed documentation, and traceability for each step and image. The package is designed in collaboration with ecologists, enabling them to create and visualise preprocessing workflows without deep technical expertise. Users can easily add, remove, and customize steps, preview results at every stage, and output fully documented workflows in a standardized format. The tool supports preprocessing of complex biodiversity image datasets collected from diverse instruments and deployment strategies. Users can apply pixel-level transformations such as colour correction and backscatter removal, and subsample data to ensure unbiased metric estimation. The software also enables integration of custom algorithms alongside built-in processing steps, enhancing flexibility. To accommodate large datasets, PAIDIVERPY includes built-in support for parallelization using Dask and integration with batch schedulers for HPC environments. Its configuration system allows users to define processing steps in a structured, interoperable format, promoting reproducibility and simplifying modifications. Extensive tutorials and documentation accompany the tool. Additionally, two curated image sets from plankton and benthic surveys, complete with metadata, are provided to support testing and exploration. PAIDIVERPY enables ecologists to focus on scientific analysis while reducing the technical burden of data preparation. It bridges the gap between ecological expertise and AI-powered image analysis by making image preprocessing accessible, efficient, and reproducible.Acknowledgements This project was supported by the UK Natural Environment Research Council (NERC) through the Tools for automating image analysis for biodiversity monitoring (AIAB) Funding Opportunity, reference code UKRI052.A recording of this session is available on YouTube: https://youtu.be/kCwyjlnridY

Full text

PAIDIVERPY: An Open-Source Python package designed to create pipelines for preprocessing image data for biodiversity analysis. Tobias Ferreira, Mojtaba Masoudi, Loic Van Audenhaege, Eric Orenstein, Colin Sauze, Jennifer Durden INTRODUCTION Standardise Preprocessing Steps Maintain Detailed Documentation Version Control & Traceability Ensure each image and preprocessing step is traceable for reproducibility. an open-source Python package designed to create pipelines for preprocessing image data for biodiversity analysis. OPEN-SOURCE DEVELOPMENT PROBLEM DISCOVERY SOLUTION IDENTIFICATION RELEASE DOCUMENTATION SCIENTIFIC IDEA CODE DESIGN DESIGN IMPLEMENTATION EXECUTION MAINTENANCE USER FEEDBACK TESTING / CI CODE REVIEW CODE QUALITY This session gives a brief overview of PAIDIVERPY’s development, highlighting the key steps in creating an open-source package. OPEN-SOURCE DEVELOPMENT PROBLEM DISCOVERY SOLUTION IDENTIFICATION RELEASE DOCUMENTATION SCIENTIFIC IDEA CODE DESIGN DESIGN IMPLEMENTATION EXECUTION MAINTENANCE USER FEEDBACK TESTING / CI CODE REVIEW CODE QUALITY PROBLEM Output evaluation Biodiversity metrics (Abundance, Diversity, etc) Training data prep AI Preprocessing BIODIVERSITY MONITORING IN IMAGERY Image preprocessing is essential to prepare data for AI models and ensuring trustworthy biodiversity metrics. Properly selecting and documenting preprocessing steps is crucial for reproducible biodiversity research. Biodiversity metrics (Abundance, Diversity, etc) PROBLEM Morris et al 2014. Limnol. Oceanogr.: Methods 12, 795–809 -Image processing workflow for highfrequency deep-sea optical imagery, enabling correction, mosaicking, and annotation of largescale datasets from AUV surveys -Goal: produce accurate, high-resolution assessments of seafloor habitats and megafaunal distribution across multiple spatial scales. PROBLEM Orenstein et al 2022. Limnol. Oceanogr.: Methods 18, 681-695 -Apply image preprocessing methods that prepare largescale plankton imagery for trait extraction and analysis -Enabling high-resolution ecological insights across diverse aquatic and terrestrial systems. PROBLEM •Camera or instrument •Sampling methods and deployment •Site conditions •Multitude of image processing approaches •Numerous hyperparameters to adjust •Handle metadata •Different tools and manual process •Not intuitive for all types of users Image preprocessing is still a challenge Every project starts from scratch and does something slightly different PROBLEM •Camera or instrument •Sampling methods and deployment •Site conditions •Multitude of image processing approaches •Numerous hyperparameters to adjust •Handle metadata •Different tools and manual process •Not intuitive for all types of users Image preprocessing is still a challenge Every project starts from scratch and does something slightly different Meshroom COLMAP CONFIGURATION FILE -YAML format -Utilizes a jsonschema file for structure, generated using Pydantic Models -Jsonschema updated automatically on the CI/CD -Easy to build tools to validate the configuration file: -Python API -Online validation tool developed specifically for this package IMPLEMENTATION IMPLEMENTATION IMAGE METADATA •Supports iFDO and CSV Files •Tools for validate and generate iFDO standard metadata IMPLEMENTATION IMAGE FORMATS AND STORAGE •Handle local and remote images (Object Store) •Image Formats in Scientific Research: -Common formats: bmp, jpg, png, tiff -Scientific imaging often involves RAW or proprietary formats •Challenges with RAW and Scientific Formats -Standard libraries like OpenCV, PIL, and imageio cannot fully handle RAW or metadata-rich images -Some RAW formats can be opened using rawpy IMPLEMENTATION IMAGE FORMATS •For others formats, it was necessary to build custom loaders •Manually parse headers, dimensions, byte order, etc •Parameters are inspired by the settings found in IrfanView PERFORMANCE AND IMAGE HANDLING IMPLEMENTATION •The images are handled as Xarray datasets (chunked or not) -Sequential Processing (n_jobs = 1) -Parallel Processing (n_jobs > 1 or n_jobs = -1) → Threads -Dask Cluster (Local or Slurm) •Code for generating benchmark calculation 300 mb dataset 4 workers / 4 threads sequential •CUSTOM ALGORITHMS IMPLEMENTATION •The user can create and use their own algorithm •Using importlib and subprocess to install dependencies and running the code TUTORIAL IN DOCUMENTATION: https://paidiverpy.readthedocs.io/en/latest/guide/custom_alg orithms/index.html UI/UX Pipeline processing -Run as a python package: -3 lines of code -EXAMPLE NOTEBOOK -Run as CLI -Run as a docker container -Images in GHCR, updated whenever a new version is deployed UI/UX •Running the package as a GUI $ paidiverpy -gui -Integrate HTML in the notebooks -Panel (HoloViz) -Convert pydantic models into forms -Integrate jupyter notebooks html into the application -The GUI is only available locally EXAMPLE OPEN-SOURCE DEVELOPMENT PROBLEM DISCOVERY SOLUTION IDENTIFICATION RELEASE DOCUMENTATION SCIENTIFIC IDEA CODE DESIGN DESIGN IMPLEMENTATION EXECUTION MAINTENANCE USER FEEDBACK TESTING / CI CODE REVIEW CODE QUALITY CODE QUALITY •TESTS -PYTEST / UNITEST -ABOVE 85% OF COVERAGE -INTEGRATE CODECOV ON THE CI FOR TEST COVERAGE •LINTING USING RUFF ON THE CI/CD MAINTENANCE AND USER FEEDBACK Workshop with end-users •Stakeholder & End-User Workshop | Mar 2025 -Ready-to-use Environment on DSP / JASMIN -Interactive Tutorials (Jupyter Notebooks) -Curated Sample Datasets •2-3 Workshops with End-Users | 2025 and 2026 ? •Working together with scientists: development of a best practices manual (not directly related to the code) https://paidiver.github.io/paidiver-workshop/ OPEN-SOURCE DEVELOPMENT PROBLEM DISCOVERY SOLUTION IDENTIFICATION RELEASE DOCUMENTATION SCIENTIFIC IDEA CODE DESIGN DESIGN IMPLEMENTATION EXECUTION MAINTENANCE USER FEEDBACK TESTING / CI CODE REVIEW CODE QUALITY NEXT STEPS •Publicise the package •Workshops with end users and test GUI •Add new preprocessing steps: feature matching, mosaicking, orthorectification, reprojection •Export metadata to AI friendly formats (such as Croissant) •Export images to other image formats, including GeoTIFF and OME-NGFF (Zarr) IMPORTANT INFORMATION AND LINKS PAIDIVERPY is ready to use. However, it is still in active development, and frequent updates and changes are expected. The API and features may evolve as we continue improving it. -GitHub: https://github.com/paidiver/paidiverpy -Pypi: https://pypi.org/project/Paidiverpy/ -Documentation: https://paidiverpy.readthedocs.io/ -ZENODO: https://zenodo.org/records/15587056 -Workshop website: https://paidiver.github.io/paidiver-workshop This project was supported by NERC through the Tools for automating image analysis for biodiversity monitoring (AIAB) Funding Opportunity, reference code UKRI052. Thank you! [email protected]