Preliminary report on progress for advanced data processing, geolocation and export format

Funded by the European Union under the Horizon Europe Programme, Grant Agreement No. 101082021 (MARCO-BOLO). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. UK participants in MARCO-BOLO are supported by the UKRI’s Horizon Europe Guarantee under the Grant No. 10068180 (MS); No. 10063994 (MBA); No. 10048178 (NOC). Deliverable 4.1 Preliminary report on progress for advanced data processing, geolocation and export format Version 1.0 31 March 2024 Julie Robidart1, Fletcher Thompson2, Patrizio Mariani2, Sarah Giering1, Motjaba Masoudi1, Carlota Muñiz3,Elisabeth Debusschere3 1 National Oceanography Centre, 2 DTU Aqua, 3 VLIZ PUBLIC Ref. Ares(2024)4567508 - 25/06/2024 2 Document Information Grant Agreement 101082021 Project Acronym MARCO-BOLO Project Title MARCO-BOLO will strengthen European marine, coastal and freshwater biodiversity observation to understand and restore ocean health. Deliverable Number D4.1 Work Package Number WP4 Deliverable Title Preliminary report on progress for advanced data processing, geolocation and export format Lead Beneficiary 28. NOC Author(s) Julie Robidart (NOC), Fletcher Thompson (DTU Aqua), Patrizio Mariani (DTU Aqua), Sarah Giering (NOC), Motjaba Masoudi (NOC), Elisabeth Debusschere (VLIZ), Carlota Muñiz (VLIZ) Due Date 01.07.2024 Submission Date DD.MM.YYYY Dissemination Level PU Type of Deliverable R: Document Version 1.0 04.05.2024, Julie Robidart Version N+1.+1 17.05.2024, Fletcher Thompson, Patrizio Mariani, Julie Robidart 18.06.2024, Sarah Giering, Motjaba Masoudi, Elisabeth Debusschere, Fletcher Thompson, Patrizio Mariani, Julie Robidart Version N+2.N+2 DD.MM.YYYY, Author 3 Executive Summary Sustainable monitoring of organisms and their habitats is imperative during the biodiversity crisis, and is especially important in marine waters where fisheries alone feed approximately 3 billion people globally while multiple threats change ecosystem dynamics. MARCO BOLO’s WP4 aims to create a direct pipeline from non-invasive, in situ monitoring of marine life, to ocean users and managers. WP4 aims to achieve this through adoption of workflows developed in WP1, FAIR data reporting, automated classification of high-volume datasets, and geolocation of sensed data in nearreal-time. The first 18 months of MARCO BOLO resulted in the development of several new deployable technologies to measure biodiversity, enabling geolocation in the field, simplicity in interacting with the software and datasets, automated classification and data processing, and enabling data flows from high-volume datasets to public repositories. While not yet field-tested, the developments described here already enable the reporting of biodiversity datasets for mapping and response, detecting ecosystems and their prey, and counting and communicating species data from the field. One publication describing these new biodiversity systems is open-access and another has been submitted. The WP4 team aims to demonstrate these developments in June, 2025 in the Belgian North Sea. This report describes the progress made in the first 18 months of MARCO BOLO WP4, Task 4.1, to “develop autonomous systems to deliver georeferenced maps of biodiversity attributes including genomic, taxonomic and habitat characteristics.” 4 Contents Document Information 2 Executive Summary 4 1. Objectives 6 1.1 Genomics 6 1.2 Particulate and plankton imaging 6 1.3 Fish and benthos 6 1.4 Bioacoustics 6 2. Genomics (Robotic Cartridge Sampling Instrument (RoCSI) and eDNA sensor) 7 3. Particulate and plankton imaging 7 4. Fish and benthos 7 5. Bioacoustics 9 6. List of potential stakeholders for each technological advancement 10 7. Coordination and planning towards the demonstration 10 Appendix 11 5 1. Objectives Work Package 4 aims “to enable new and advanced technologies for cost-effective, timely and accurate biodiversity observations in coastal and marine regions.” This Report addresses progress specifically in Task 4.1, to “develop autonomous systems to deliver georeferenced maps of biodiversity attributes including genomic, taxonomic and habitat characteristics.” Deliverable 4.1 is achieved through 7 complementary technologies targeting diverse biodiversity variables. Genomics technologies (a sampler and sensor) are upgraded to facilitate data delivery and usability, targeting biodiversity using eDNA. Particle and plankton images are more quickly identified using machine learning algorithms based on the Underwater Vision Profiler (UVP-6). An integrated multi-beam echosounder and camera system on an Remotely Operated Vehicle (ROV) are used to map the benthos, while creating mosaics of the seafloor. These mosaics are processed to automatically identify and quantify whichever benthic species are present, such as starfish and fish. Last, methodologies are developed to process large-volume acoustics data to quantify fish biomass and detect marine mammal echolocation clicks, using an echosounder and the C-POD, while detecting tagged fish with an acoustic receiver. 1.1 Genomics The genomics subtask aims to add geolocation information to the automated genomic sampler, RoCSI (the Robotic Cartridge Sampling Instrument). It will additionally enable automated acquisition and processing of fluorescent probe data from the genetic sensor, and communicate these data to the user. 1.2 Particulate and plankton imaging This subtask aims to integrate the UVP6 into the autonomous vehicle and develop onboard image processing capabilities that use a “miniaturized AI system” for real-time image classification directly within the vehicle. The on-board classification results will be summarized and transmitted over-thehorizon along with geolocation data. 1.3 Fish and benthos The aim of this subtask is to map the benthos and pelagic fish incorporates Ultra Short Base Line acoustics for positioning of an open source BlueROV2 Remotely Operated Vehicle (Blue Robotics) integrated with a new high-definition multi-camera system. Images will be processed to provide data products in the form of processed large-scale seafloor images and annotations of identified animals. 1.4 Bioacoustics Monitoring the marine ecosystem through acoustics is considered a non-intrusive method. A standalone mooring is developed to accommodate multiple (acoustic) sensors for long-term recording at sea. An acoustic fish receiver detects tagged fish, a broadband hydrophone listens to the acoustic environment of marine fauna and a POD registers echolocation of marine mammals. The combination of detecting marine mammals, their prey and the acoustic environment is important, towards an ecosystem approach. A data pipeline is developed for CPOD/FPOD data to (1) detect and classify echolocation click trains from vocalizing harbour porpoises (Phocoena phocoena), (2) export a yearly dataset on hourly resolution to the Emodnet Biology and Eurobis, according to the FAIR principles. A standardize dataflow is currently being developed for the scientific echosounders. 6 2. Genomics (Robotic Cartridge Sampling Instrument (RoCSI) and eDNA sensor) 2.1 Robotic Cartridge Sampling Instrument (RoCSI) - description The RoCSI (Figure 3.1) is an automated ‘omics and eDNA sampler that preserves samples in situ. It filters up to 4L seawater onto >0.2 um commercial Sterivex filters, and performs a bleach flush to decontaminate. The user can create a mission schedule or sample continuously, programmable by a simple Graphical User Interface (GUI). Samples can be processed in the lab for ‘omics to learn about taxonomy and function or environmental DNA to get a fingerprint of local biodiversity. RoCSI has been pressure tested to 6000 m and deployed to a maximum of 4719 m depth. RoCSI has pressure sensors to prevent clogging and flow meters to measure the volume filtered. The output .csv file reports each of these parameters, the sample identification and the local time for each filtration event. After recovering RoCSI from fieldwork, scientists are tasked with merging the time stamp from this output report with the vehicle or ship’s GPS data to map each sampling event and integrate datasets with other sensors’ datasets. Automation of this process would save time and prevent human error. It would also allow easier follow-up investigation when rapid response is required. 2.2 Progress towards RoCSI geolocation in the field NOC have evaluated several options for geolocation of RoCSI in the field. The simplest solution would be to add GPS to the RoCSI instrument, and communicate those data through the software for reporting in the sample output file. This simple solution would be easily used by RoCSI wherever it might be. However, for most marine operations the RoCSI data will have to be synced to colocated sample / sensor data. Two sources of location data, which will never be perfectly aligned, can complicate this as well as data reporting. Scientists at NOC have determined that the primary source for geolocation should be the platform or vehicle (e.g. ship or autonomous vehicle) and that these data should be harvested by RoCSI at the onset of each sampling event. Research vessels currently communicate location information periodically to instruments that are networked on the ship, and those instruments incorporate this universal location data so that they are synced in both time and space. NOC are developing software to harvest these data and report them in the RoCSI sampling data output file as our priority (Objective 1) and are considering incorporating GPS into the RoCSI itself (Objective 2), for cases where instrumentation does not have a primary “home” (i.e. a single RoCSI that is reused across many vehicles and platforms, or ships of opportunity). 2.3 Progress towards RoCSI data bundling and communications NOC have developed software to reconcile the RoCSI output data (Time, Cartridge ID, sampling duration, Treatment (preservation), Stop Reason, Volume Seawater Pumped, Maximum Pressure) with the geolocation data from the Autosub Long Range (ALR5): Latitude, Longitude and Depth, into a single output .csv file, using data from TechOceanS RoCSI deployments in Gran Canaria in Figure 3.1 | The Robotic Cartridge Sampling Instrument, or RoCSI, is an automated genomic sampler. 7 March 2024. The plan is to communicate this .csv format over radio or iridium channels from a mobile platform. 2.4 eDNA sensor description A genomic sensor, the LAMPTRON, is a device developed at the NOC that holds a molecular reaction at a stable temperature while emitting excitation wavelengths specific for DNAor RNAprobes and interrogating optical signals using FAM and Sybr fluorescence (excitation-emission spectra). It was designed for isothermal amplification of DNA or RNA using isothermal analytical chemistries (Recombinase Polymerase Amplification or RPA or Loop-mediated Isothermal Amplification or LAMP), with gene-specific probes or with Sybr, which binds to all double-stranded DNA. 2.5 Progress towards eDNA sensor data analytics and reporting A new chip was designed with a black tint to decrease the background brightness of the analytical cell, thus decreasing the limit of detection of the fluorescent signal. The optical output is now automated, with an excitation pulse and the reaction’s emission reading taking place and logged every 100 milliseconds, then averaged to report a reading every 30 seconds without user interaction. Post-reaction, total data are visualised through the GUI after normalisation to background fluorescence over the entire reaction, in an intuitive format for molecular biologists, similar to the output from quantitative PCR lab instrumentation. 2.6 Genomics technologies plans and timelines RoCSI The work to resolve vehicle geolocation data with RoCSI output data files uses geolocation from the Autosub Long Range vehicle, which is available in standard .csv file format. This work leveraged fieldwork planned as part of EU Project TechOceanS. However, MARCO BOLO objectives utilise a surface vehicle more suitable for coastal operations and NOC are currently evaluating which vehicle this might be (the C-worker, planned for MARCO BOLO demonstrations at the proposal stage, is not operational). Current plans are: 1) define the vehicle for the demonstration (by August 2024), 2) request geolocation data format and current communication protocols, if they exist (August 2024) and 3) develop software for RoCSI to collect these data and generate an integrated output file. In parallel, geolocation capabilities will be developed for RoCSI operations in a stand-alone mode (i.e. for universal field operations, expected July 2025). Genomic sensor The next steps are to automate the visualisation of amplification curves of all standards on a single graph, and the processing of data from the standards to create a standard curve. Next is automation of the calculation of gene copy numbers from that standard curve (estimated delivery October 2024). NOC have shared representative data with Task 4.3 leads in order to begin the process of networking and reporting these data over user-friendly graphical interfaces from the field. This has to occur after the communications interface and protocol are decided for the final demonstration (Task 4.4), with an estimated delivery of June 2025. 8 3. Particulate and plankton imaging 3.1 Particulate and plankton imaging with the UVP6 The UVP6, a specialized underwater imaging device for integration on autonomous platforms, captures high-resolution images of particulate matter and plankton as it travels through the water column. Each image is processed within the camera hardware to produce ‘vignettes’ that feature a singular object centered within the frame. Currently, on board, the UVP6 extracts simple ‘hardcoded’ features and uses these for a rough classification of the objects in the vignettes. This rough classification is saved on the device and not accessible until the device is retrieved. For autonomous vehicles, this poses a big problem as they are frequently not recovered. UVP6 advancement as part of MARCO BOLO involves developing a workflow that allows the classification of images on board using a much improved, ‘smarter’ algorithm and that enables near-real-time awareness of the ecosystem, ensuring maximal data exploration without the need to retrieve the vehicle. Figure 3.2 | Current workflow of image processing in the UVP6 (shaded in blue) and advances carried out in MARCO-BOLO (shaded green). 3.2 Progress towards particle and plankton imaging data processing Advancements in machine learning integration with the UVP system have been significant. The current workflow has been mapped and the best access point in the current software and hardware configurations has been identified (see Figure 3.2). NOC’s training dataset for the development of this workflow includes 63,500 annotated images. The geographical locations of this data set cover a broad range of oceanic regions, including the Mediterranean Sea, equatorial Atlantic Ocean, equatorial Pacific Ocean and polar regions. The data set comprises images from 980 dives between 0 and 5000 m depth, with the highest data density within the upper 1000 m of depth (Figure 3.3). This dataset is instrumental in training NOC’s machine learning models, ensuring high accuracy in identifying and classifying marine organisms based on their unique characteristics. 9 Figure 3.3 | Location, depth, scale(size) range of images used for training the new CNN classifier, highlighting the broad geographical coverage. In addition, a convolutional neural network (CNN) has been successfully integrated, on a low-power Raspberry Pi, enabling the classification of images into distinct categories such as species and size. This new CNN classifier has an improved classification accuracy (+24 %points) compared to the current on-board classifier using the hard-coded feature extractions (Table 3.1). To evaluate the performance of the existing UVP6 classifier against this new deep learning approach, the feature extraction and classification framework used by the current UVP6 classifier was precisely replicated, to guarantee a fair and unbiased comparison. The performance of the current UVP6 classifier was tested against NOC’s novel deep feature extractor across three different configurations: ● Current UVP6 classifier (Base model: B), ● NOC’s feature extractor method with a 16-dimensional latent space (Model 1: M1), ● NOC’s feature extractor method with a 55-dimensional latent space, equivalent to the feature space of the current classifier (Model 2: M2). NOC designed a routine that processes data in batches and produces a summary of classifications in a format compatible with the platform's communication protocol for transmission via satellite. This method ensures timely and efficient data transmission, essential for ongoing marine research. 16 Figure 3.8 | a) Nephrops sledge, b) Operator screen, c) Marked objects: Red: Burrows, Yellow: Animals 4.5.6 Refinement of multi-camera acquisition When purchased, the Blue Atlas camera system was configured to record video from all attached cameras using h264/h265 compression which was not configurable. This was not ideal as compression elements were present in the images due to too high compression factor settings which could not be adjusted. In an underwater environment it is especially easy for these artifacts to become apparent as the low contrast and brightness of many underwater scenes produce blocky low-quality compressed images. A new mode was added to the system by DTU technicians to enable the system to record lossless compressed images at a specified frequency, as well as an alternative h265 video compression script with adjustable encoder parameters. The new modes ensure better results from 3D reconstruction as there are less artifacts in the captured data. 17 Figure 3.9 | A) 3D reconstruction of the surface of a cross in ASTA (Autonomous Systems Testing Arena), created using images captured from a multi-camera system. B-E) Subset of images used in the reconstruction, taken at the same time-stamp for cameras 1-4; An additional camera used in the reconstruction is omitted. Figure 3.10 | A) 3D point cloud of a Pike captured in Skovshoved Havn, Denmark. Surface reconstruction of seaweed is a work in progress. Created using images captured from a multicamera system, B-E) subset of images used in the reconstruction, taken at the same time-stamp for cameras 1-4, An additional camera used in the reconstruction is omitted. 4.6 Fish and benthos detection plans and timelines With the platform integrations largely completed, the systems are ready for testing in short fieldwork campaigns in Danish waters for 2024. As such, both ROV platforms will next be deployed in June 2024 as part of regular fieldwork activities in the Øressund and Køge bay regions. Here, they will acquire data of the sea bottom as part of bottom classification and impact studies. The data will be used to identify fish species (such as cod and flounder) that are residing close to the bottom 18 and cannot be identified by traditional fisheries acoustic methods, as well as produce georeferenced maps of the areas surveyed. Plans for 2025 fieldwork with VLIZ in the Belgian North Sea have been made. The developments in the data analysis pipeline made from the fieldwork completed in 2024 will be then further refined after the 2025 data acquisition campaign. 5. Bioacoustics 5.1 Bioacoustics for the detection of marine mammals – description Harbour porpoise (Phocoena phocoena) are the most common marine mammals in the Southern North sea. Porpoises as well as dolphins (Odontoceti) use echolocation to extract information from their surroundings. Dolphins produce clicks in a wide frequency range and are typically short and loud while porpoises produce longer and weaker clicks in a narrow frequency range (120 - 145 kHz, mode 132 kHz). These clicks can be recorded by the passive acoustic device, C-POD or F-POD (Chelonia Limited, Figure 3.11), when a marine mammal is swimming in the vicinity of the recorder. The POD can record clicks between 20 and 160 kHz including ambient background noise, sonar and other biotic underwater sound. The key to the performance of the C-POD is detection and classification of series of clicks, so-called click trains. Click trains have distinctive features which are used by the classification algorithms to identify the occurring cetacean species. The output of these sensors result in absence and presence data per minute of vocalizing porpoises. Figure 3.11 | Example of a click train of harbour porpoise recorded by a C-POD in the Belgian part of the North Sea (Chelonia Limited). 5.2 Progress towards multi-sensor mooring for biological observations The multi-purpose mooring is equipped with a broadband hydrophone, cetacean logger as well as an acoustic fish telemetry receiver. The dataflow of the cetacean logger is described in Figure 3.12, from Calonge et al., submitted. A data paper has been submitted describing the entire data flow using CPOD data, from collection of echolocation trains of porpoises, processing of the data and flow to Emodnet Biology and Eurobis international data portals, according to the FAIR data principles. The combination of two technologies, cetacean logger and acoustic telemetry, has proven its value to study the co-occurrence of species over time and space (Calonge et al., 2024). 19 Figure 3.12 | from Calonge et al., submitted. Schematic overview to obtain and maintain the harbor porpoise data series from data acquisition to harvesting of biodiversity information (solid arrows), and the data files involved in each step (broken arrows). Data read from the PAM loggers (DATA0.CHE, DATA1.CHE, ...) are developed into .CP1 files, classified and manually validated as .CP3 files, and exported as (1) Detections and environment and (2) Train duration one-minute resolution text files according to quality class. The text files as well as the deployment metadata are uploaded on the European Tracking Network (ETN; http://lifewatch.be/etn) database, which could be visualized and analyzed through the LifeWatch data explorer (http://rshiny.lifewatch.be/cpoddata/) and the lwdata package accessible on rstudio.lifewatch.be. Datasets in minuteand hourlyresolution are both published yearly with a Digital Object Identifier (DOI) on the Integrated Marine Information System (IMIS; https://www.vliz.be/en/imis) and Marine Data Archive (MDA; https://marinedataarchive.org/). Datasets in hour-resolution, aggregated from the minuteresolution datasets, are published in a Darwin Core Archive format (DwC-A) on IMIS and in several unrestrictive repositories. 5.3 Bioacoustics data analyses plans and timelines The co-collection of acoustic data to classify cetaceans and fish continues, using the multi-sensor mooring in the North Sea. The established data flows will enable further dissemination of highresolution data into public repositories. The processes described above will be further validated for North Sea testing in June 2025. 6. Potential stakeholders for each technological advancement 20 WP4 participants have determined data types of potential value to stakeholders, including: • maps of biodiversity and target species (collected via sensor-platform combinations), • maps of phytoplankton biodiversity scaled up regionally using satellite datasets (mapped Essential Biodiversity Variables, linked with drivers and pressures of biodiversity change), • easy-to-use data products (e.g. number of starfish and GPS location) and • smarter networked sensors (responding to key remotely-sensed variables like chlorophyll). Milestone 4.1, Stakeholder presentation on new observing tools and methodologies: Stakeholder engagement has begun via a presentation of smartphone-reported bird song analyses by the University of Seville at the 1st COP and Co-Design Workshop on 23 May, 2024. WP4 plans to engage further with representations of the datasets above at the next COP and CoDesign Workshop in the Autumn, 2024. The University of Seville has representative data types from WP4 sensors, and will develop reporting systems using these, based on needs identified at the upcoming co-design workshop. Stakeholders represented at the first co-design workshop include policymakers, and WP4 scientists have additionally identified marine managers, offshore and sustainable marine monitoring industries as potential WP4 data users. These sectors will be engaged in co-design through future meetings (by January 2025). 7. Coordination and planning towards the demonstration WP4 has created a plan for the North Sea demonstration, tentatively planned for June 2025. Work will take place at the Grafton site in the Belgian North Sea. Operations will take place a the Grafton site in the Belgian North Sea. A lander will include echosounders (fish schools), C-PODS and F-Pods (porpoises dolphins, toothed whales), acoustic receivers (tagged fish), an eDNA sampler (metazoan biodiversity analyses), ADCP (currents), CTD (salinity, temperature and depth) and a turbidity sensor. VLIZ has designed and manufactured a larger lander to accommodate all the sensors listed here. The BlueROV2 will be operated via the RV Simon Stevin, mapping the benthos in the vicinity of the landers (using USBL, imaging and 3D mosaics), and including CTD and turbidity sensors. An uncrewed surface vehicle near the Grafton site will include the UVP6 (particulate and plankton imaging and classification), an eDNA sampler (plankton biodiversity analyses and activity), CTD, turbidity and potentially low-level nutrient concentrations. In preparation for field trials, VLIZ have assembled map layers using a new Python package including seabed habitats, bathymetry and wave formation https://github.com/lifewatch/bpnsdata. They’ve also supplied maps and datasets of regional turbidity, marine obstacles such as seafloor cables and pipelines, and nautical information to plan operations. WP4 has been meeting monthly or more to plan for the June 2025 (as currently planned) field demonstration. 21 1) Calonge A, Goossens J, Muñiz C, Reubens J, Debusschere E (2024) Importance of multi-sensor observations to advance species co-occurrence knowledge: a demonstration of two acoustic technologies. Mar Ecol Prog Ser 727:49-65. https://doi.org/10.3354/meps14496 Funded by the European Union under the Horizon Europe Programme, Grant Agreement No. 101082021 (MARCO-BOLO). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. UK participants in MARCO-BOLO are supported by the UKRI’s Horizon Europe Guarantee under the Grant No. 10068180 (MS); No. 10063994 (MBA); No. 10048178 (NOC). Project Coordinator Nicolas Pade | [email protected] Project Manager Giulia Vecchi | [email protected] Press and Communications Mathilde Vidal | mathi[email protected] Website: MarcoBolo-Project.eu Twitter: @MARCOBOLO_EU LinkedIn: MARCO-BOLO