A Machine Learning Application to Camera-Traps: Robust Species Interactions Datasets for Analysis of Mutualistic Networks

! Camera trap protocols for animal-plant interactions. V. 1.1 - Villalva & Jordano! AI-based workflow for recording animal-plant interactions data with camera traps Pablo Villalva & Pedro Jordano with the collaboration of: Francisco Rodríguez-Sánchez, Eva Moracho, Jorge Isla, Elena Quintero and additional help with field work from Gemma Calvo and Pablo Homet.! Contact address: Integrative Ecology Group Estación Biológica de Doñana, EBD-CSIC Avda. Americo Vespucio 26 E-41092 Sevilla, Spain Voice: +34 95 4466700 fax: +34 95 4621125 E-mail: [email protected], [email protected] http://pjordanolab.ebd.csic.es A separate GitHub repository includes detailed scripts for this protocols. https://github.com/Cyanopica/Ecological-interactions-camtrap-protocol Version 1.1. May 2023 This study was funded by MICINN through European Regional Development Fund [SUMHAL, LIFEWATCH-2019-09-CSIC-4, POPE 2014-2020] ], CSIC Interdisciplinary Thematic Platform (PTI) Síntesis de Datos de Ecosistemas y Biodiversidad (PTIECOBIODIV). ! Index A. Introduction 3 - 6 !A.1 Study area 3 "A.2 Study species 4 "A.3 General approach 5 "A.4 Video data workflow outline 6 B. Pre-processing 6 - 12 "B.1 Database structure 7 ""B.1.1 Deployments 7 ""B.1.2 Videos 8 ""B.1.3 Observations 9 "B.2 Camera trap settings 9 "B.3 Video dumping and data storage 10 C. Processing 12 - 17 "C.1 Eliminating empty images 13 "C.2 AI implementation 13 "C.3 Video splitting 14 "C.4 Running the model 14 "C.5 Model output 15 "C.6 Confidence selection 16 "C.7 Visualisation and database creation 16 D. Post-processing 17 - 19 D.1 Database handling 17 "D.2 AI performance metrics 18 2 A. Introduction This is a summary of our protocol for processing large numbers of video-recorded file sets generated during camera trap surveys for monitoring animal-plant interactions (i.e visits of frugivores to fruiting plants). We combine a camera trap field protocol (preprocessing) with several tools for time-saving processing (and post processing). This workflow enables us to manage camera traps in the field, transfer and storing data to the lab and post-process large video files batches, reducing effort and time in the database compilation. The method combines a field protocol based on camera trap operation and data standards together with Artificial Intelligence for image recognition and a viewer program to view and tag images. The main objective is to build an accurate and fully annotated dataset for animal-plant interactions records while time effort is minimised.! A.1 Study area Doñana National Park is a unique protected area located in Huelva, SW Spain. It is characterized by a large variety of terrestrial and aquatic ecosystems ranging from pine and cork oak forests to scrublands, grassland, sand dunes, and marshlands. The rich diversity of ecosystems is the main reason for harbouring a great biodiversity, evidenced by more than 1300 plant species (170 of which are endemic), over 300 bird species and 50 mammal species, including emblematic species such as the imperial eagle and Iberian lynx (Green et al. 2016). ! Plant species have a crucial role in maintaining the mentioned animal diversity through bottom up processes, while herbivores control vegetation through top down regulation. However animals not only maintain plant communities by freely eating upon them, but offer an important ecosystem service as dispersal vectors for a variety of plant species. The offer of fleshy fruit for seed dispersion is a common evolutionary strategy in Doñana in the so called endozoochorous dispersal syndrome. In fact 56 % of woody species in Spanish Mediterranean scrublands are adapted to endozoochorous seed dispersal by vertebrates (CITAS: Herrera 84, Jordano 84), becoming a central process in plant populations where natural regeneration strongly depends upon seed dissemination by animals (Jordano, 2000).! Frugivorous birds and mammals visiting fleshy fruit trees and scrubs in Doñana may behave as seed dispersers, pulp consumers, or seed predators. Even though the mutualistic-antagonistic continuum is the rule, some species such as Sus scrofa or Chloris chloris are prominently seed predators, while others such as Vulpes vulpes and Erithacus rubecula can be considered fully legitimate seed dispersers. ! The local plant populations studied here are located in the Doñana Biological Reserve, a core area within Doñana National Park. In this area fleshy fruit species are spread throughout the landscape generally occurring in isolated patches, some species may be more continuous across the landscape such as Olea europaea and some are associated with ecotone areas, such as Rubus ulmifolius while others are more associated to a specific type of soil such as Corema album that occupies coastal dunes. ! 3 A.2 Study species The study species correspond to the most relevant plant species producing fleshy fruits in Doñana National Park (SW Spain). The selected twelve species are listed and briefly described below: ! - Corema album is a shrub endemic from the west coast of the Iberian Peninsula, and the Azores Islands growing mainly on sand dunes. It is considered an endangered species (Ortiz et al., 1998) that has experienced a notable decline in size and number of populations. It is a dioecious shrub that rarely exceeds 1 m height. It is wind pollinated and its fruits are quasi-spherical, white drupes. The pulp has a high water and sugar content (Calviño-Cancela, 2003; Herrera, 1987). Ripe fruits are available in summer and early autumn, with the highest availability occurring in July - August.! - Juniperus phoenicea is a gymnosperm shrub inhabiting coastal dunes and rocky habitats in the western Mediterranean and Macronesian archipelagos. It is an anemophilous species characterized by masting cycles of fleshy cone (galbule) production (Jordano,$1993; Roques et al.,$1984). Brown-red galbules are consumed and dispersed by several thrush species and medium-sized generalist mammals. The fruiting period is in autumn, spanning from October - December.! - Juniperus oxycedrus subsp. macrocarpa is a gymnosperm growing in the northern mediterranean basin and northern Africa. It is a dioecious and anemophilous species that produces berry-like spherical, fleshy cones. Unripe cones are green, ripening in 18 months when they turn to orange-red with a variable pink waxy coating that are available in winter and early spring, between December - March. ! - Rubus ulmifolius is a rosaceous vine-shrub native across Western Europe and naturalized in N. America, NW and S. Africa and Australasia. It is unique among subgenus Rubus in displaying sexual entomophilous reproduction while all others are facultative apomicts. The fruit is a polidrupe, dark purple, almost black with a summer fruiting season that expands from July - August.! - Pyrus bourgaeana is a rosaceous tree, widely distributed across the southern Iberian Peninsula and northern Morrocco but with fragmented populations that occur at low densities and small patches. It is pollinated by insects and its fruits are non-dehiscent globose pomes with green or brown skin inconspicuous to birds and a styptic pulp. The autumn fruiting season expands from September - November.! - Smilax aspera is a perennial, evergreen climber with a flexible and delicate stem, with sharp thorns from the family Smilacaceae, widespread in Africa, Europe and temperate and tropical Asia. The entomophilous flowers are very fragrant, and the fruits are globose berries, gathered in clusters that are initially red, later turn black and have an extended phenology from October - March. ! 4 - Myrtus communis is an evergreen shrub or small tree from the family of Myrtaceae native to southern Europe, North Africa, Asia and Macaronesia. Reproduction is entomophilous and the fruit is a blue-colored fleshy berry when ripe. Fruiting occurs in late winter from December - February. ! - Arbutus unedo is an evergreen shrub or small tree in the family Ericaceae, native to the Mediterranean region W Europe. The entomophilous hermaphodite flowers are white and hang from a reddish panicle. The fruit is a red, spherical berry with a rough surface. Fruit ripening occurs in about 12 months after anthesis, at the same time as the next flowering but with a fast ripening between December - January.! - Olea europaea, var. sylvestris is a species of evergreen tree or shrub native to Mediterranean Europe, Asia, and Africa in the family Oleaceae. Pollination is anemophilous and the fruits are small drupes black when ripe, thinner-fleshed and smaller in plants of this wild subspecies than in orchard cultivars of olive trees. Fruiting (fruit ripening) occurs from November - December.! - Asparagus aphyllus is a dioecious species, climbing plant in the family Asparagaceae native from the Mediterranean basin. Flowers are pollinated by insects and fruits are globose dark-green berries that torn blackish when ripe. Fruiting occurs in autumn or early winter, between October - December.! - Rubia peregrina is a herbaceous perennial plant species belonging to the family Rubiaceae mainly present in the Mediterranean basin, Great Britain and North Africa. The hermaphroditic flowers are pollinated by insects. The fruits are fleshy green berries, black when ripe which occurs in late summer and early autumn between September - November.! - Osyris lanceolata is a a hemiparasitic evergreen shrub in the family of Santalaceae found in low densities across Africa and the southern half of the Iberian Peninsula and Macaronesia. They are self-fertile, so the species produces fertile seeds prolifically. The fruit is a single spheroid colorful drupe, progressing from greenish tones to bright orange as they ripen. The species produces ripe fruits almost continuously, and most individuals have fruiting periods virtually encompassing the entire year. The peak of the fruiting period of individual plants may occur in almost any month of the year.! A.3 General approach Our approach for monitoring animal-plant interactions in natural habitats involves the strategic placement of camera traps, aimed at specific plant species, referred to as focal plants. These cameras are set in video mode, providing us with valuable insights for species identification and behavior, as well as an accurate quantification of fruit consumption.! Our video-based monitoring method offers several advantages over traditional camera-trap techniques based in still pictures. By capturing movement, we are able to identify animal species and their behavior, allowing us to gain a deeper understanding of the complex interactions between animal and plant species. Additionally, by accurately measuring fruit consumption at least in some video recordings, we can determine consumption rates and fruit feeding behavior, 5 and gain insights into the importance of different plant species for wildlife in the ecosystem as well as for the ecosystem service provided by animals for plant dispersal.! However, camera-trap monitoring can also present challenges, especially in environments with high wind levels. In these conditions, incorrect triggering can easily occur through the movement of grasses and tree branches, leading to a large number of empty images. To mitigate this issue, we have developed a protocol that streamlines the process of generating large databases from video recordings and reduces the time and effort required to do so.! This camera-trap monitoring approach provides a powerful tool for studying plant-animal interactions in natural habitats. By accurately identifying species and capturing animal behavior, as well as fruit consumption rates, we can gain valuable insights into the functioning of complex ecosystems.! A.4 Video data workflow outline B. Pre-processing Most likely a camera trap field sampling for ecological interactions involves the simultaneous deployment of multiple cameras, in replicated positions to target different individual plants, throughout the fruiting season of different focal plant species. The cameras are checked at regular intervals, typically weekly, biweekly, or monthly, which can result in a large number of videos with the same name and date.! 6 Figure 1. Summary of the protocol streamline. Effective management of such a large and complex data set is crucial for a successful database creation. To achieve this, a structured field database is required to keep track of the data at every stage of the process. For this purpose, we recommend the use of the Camera Trap Data Package (Camtrap DP), a community-developed data exchange format that is under development as a Biodiversity Information Standard (TDWG). This package provides a useful structure for controlling camera-trap data at three levels, from which we will adopt the structure: deployments, revisions, and observations.! Camtrap DP offers a standardized format for organizing camera-trap data, ensuring consistency and reducing the risk of errors or inconsistencies. This structure includes all necessary information, such as camera settings, deployment locations, and video file names, allowing for easy management and analysis of the data. A template for the Camtrap DP is available for use, and you can find a template for our ad-hoc structure in the GitHub repository https://github.com/ Cyanopica/Ecological-interactions-camtrap-protocol/tree/main/Preprocess or see the following descriptions.! The main data is structured in three related plain text files (.csv) as follows:! B.1 Database structure B.1.1 Deployments Table with camera trap deployments. Includes deploymentID (focal species acronym + cameraID), Location and camera Setup information for each camera.! File Description deployments.csv Table with camera trap deployments. video.csv Table with media files captured by the camera traps. observations.csv Table with observations based on media files (after viewing in Timelapse) Name Definition Type Deployment_ID Unique identifier of the deployment. Name of the focal species followed by the individual number low dash camera number. Example: Aune003_58 string Location Name given to the deployment location. Survey area. string Longitude Longitude of the deployment location in decimal degrees, using the WGS84 datum. number Latitude Latitude of the deployment location in decimal degrees, using the WGS84 datum. number Start Date and time at which the deployment was started. Formatted as an ISO 8601 string with timezone designator (YYYY-MM-DDThh:mm:ss±hh:mm). datetime 7 ! B.1.2 Videos Table with video files captured by camera traps. Associated with deployments (by deploymentID) and organised in revisions (revision_ID). Includes Timestamp_Issues and File_path.! End Date and time at which the deployment was ended. Formatted as an ISO 8601 string with timezone designator (YYYY-MM-DDThh:mm:ss±hh:mm). datetime Days Number of days the deployment was set in the field. End_date - Start_date number Setup_by Name(s) or unique identifier of the person that deployed the camera. string Camera_ID Unique identifier of the camera used for the deployment (could be the serial number but also a simple number) string Camera_model Manufacturer and model of the camera. string Comments Comments or notes about the deployment. string Name Definition Type Deployment_ID Unique identifier of the deployment the media file belongs to. Foreign key to Deployments.Deployment_ID. string Revision_ID Unique identifier of the revision the media file belongs to. Revisions contain one or more media files (e.g. a single image or video or a sequence of successive images or videos). Example: Rev_01 string Videos Number of media files. number First_video Datetime for the first video in the revision sequence. datetime Last_video Datetime for the last video in the revision sequence. datetime Days Number of days that the deployment was set in the field during the current revision. number Setup_date Date at which the camera was set on in the current revision. datetime Revision_date Date at which the camera was set off in the current revision. datetime Functioning_days Number of days where the camera was functioning. Revision_date - Setup_date number Battery Percentage of battery in the revision datetime. string Timestamp_Issues True if timestamp in the media have been detected. boolean File_path URL or relative path to the media files, respectively for externally hosted files. string Favourite True if it contains videos tagged as favorite. boolean 8 B.1.3 Observations Table with video files results from visualization. Associated with deployments (deploymentID) and with revisions (revision_ID) through Videos.file_path. Note that this data will be generated directly with Timelapse software as explained below. ! Comments Comments or notes about the revision. string Name Definition Type File Name of the video file. If more than one video files use concatenate separated by “,”. string Path URL or Relative path to the first Obs.File, respectively for externally hosted files. string Plant_sp Name of the focal plant species. string Plant_ID Unique identifier of the plant individual. Example: Sasp003 string DateTime Date and time at which the video started. Formatted as an ISO 8601 string with timezone designator (YYYY-MMDDThh:mm:ss±hh:mm). datetime Sp1 Latin binomial for the principal animal species recorded in the video. Example: Athene noctua. string Behaviour string Sp2 Latin binomial for a secondary animal species recorded in the video. string Behaviour_Sp2 string Sp3 Latin binomial for a third animal species recorded in the video. string Behaviour_Sp3 string n_cam Number of cameras set in the same focus individual. number Videos Number of videos recorded in the current revision. Vid.Videos number Days Number of days for the camera recording in the current revision. Sampling effort for a video in a given revision. number Video_events When videos are collapsed in time (i.e 5 minutes) number of videos that where collapsed. It will be coincident with the number of different Obs.File names. number Duration Number of second for the duration of the event. Could be the sum of durations from different Obs.video_events. number TimeStamp_Issue True if timestamp in the video have been detected. Boolean Long Longitude of the deployment location in decimal degrees, using the WGS84 datum. number Lat Latitude of the deployment location in decimal degrees, using the WGS84 datum. number 9 C.6 Confidence selection After the video-level output is generated, it is imperative to incorporate it into the workflow in an efficient manner. One option is to load the AI results into a Timelapse software, but this approach may pose challenges when dealing with large video-level results. As an alternative, a confidence threshold can be employed to select only those videos that meet the required criteria, while ignoring the others. This approach, while not very flexible, enables stepwise selection of confidence intervals in successive rounds. For instance, it is recommended to start with a higher confidence threshold (e.g. above 0.8) and then move on to lower confidence ranges (e.g. from 0.7 to 0.8) in later reviews. However, it is important to note that different versions of MD can have varying confidence profiles, so the optimal threshold values may vary greatly. For example, in our dataset, the confidence threshold was set to 0.8 when using MD version 4, while it was set to 0.15 for MD version 5.! In order to select and manipulate files, the file.copy function from base R was used as a low-level interface to the computer's file system to copy and paste the selected videos. A script, which includes the creation of destination folders, selection of file lists, and copy and paste function, can be accessed via https://github.com/Cyanopica/Ecological-interactions-camtrap-protocol/tree/ main/Process. ! C.7 Visualisation and database creation! Table 3. JSON structure for a MD output at video level. Categories, version information and results from image detection are shown. Note that file shows the relative path to the video file, $detections shows confidence level for each frame as a list and $max_detection_conf shows the maximum value for $detections. Categories Example $detection_categories [1] animal "animal" [2] person “person" [3] vehicle “vehicle" $info $detection_completion_time 2023-01-10 02:26:23 $format_version info.detector 1.2 md_v5a.0.0.pt $detector_metadata v5a.0.0 $images $file Rubus/Rev11_20220831/Rulm015_15/IMG_0035.AVI $detections 1, 0.888, 0.256, 0.18, 0.155, 0.155 $max_detection_conf 0.888 16 We used Timelapse an open-source tool for reviewing camera trap images and videos. It boasts good support for videos, and a multitude of interface tools for accelerating the visual analysis and encoding. The program automatically extracts file information and metadata, presenting a custom interface for data entry and supporting visual searches. All the data will be saved to a CSV file. If you need help using Timelapse, you can find the reference guide here.! To get started, you'll need to create a template specifically for your project using the template manager. This will allow you to gather data from each image. For gathering animal-plant interactions, the template should match the structure of the "Observations.csv" file explained in the preprocessing section, as it will be automatically generated once you've finished viewing and annotating the video set.! Once your template is set up, load the video files into Timelapse, making sure to preserve the relative path structure. Then, visualize the video set and make note of at least the animal species and its behavior.! Here's a list of example behaviors that we recorded for our dataset:! 1. Eating.! 2. Probably eating. ! 3. Searching for food.! 4. Visiting (using plant).! 5. Walking (or flying by the image).! 6. Others (note in observations).! 17 Figure 4. Time-lapse user interface. D. Post-processing D.1 Database handling Once the video-level database is generated, it is necessary to have a clear understanding of the analysis that will be carried out to handle the information. It is important to note that the most refined data for the dataset is at the video level. This means that each entry represents a 10 second duration video (although duration may be longer depending on the camera setup) with different animal species exhibiting different behaviors. First steps for managing this data will require to select the desired behaviors which in our frugivore context should be related to fruit consumption. If a conservative approach is desired, it is possible to select only those videos where the animal was found ingesting the fruit. However, this type of selection is likely to be unrealistic and too strict, actually underestimating the frequency of fruit feeding within a specific visit to the plants. Adding those videos where the animals are probably eating (videos where the animal was not recorded ingesting due to its position or maybe has allegedly been eating out of shot) would be more realistic. However this approach may also under-estimate the real amount of frugivore events as fruit-eating events may not be captured within the camera shot. Including videos where the animals are in a food-seeking attitude under or on top the focal plant is the most realistic approach to record animal-plant interactions for this protocol.! Once behaviors are identified and selected, it is recommended to establish a baseline for defining the events, as if we were to keep the data at the video level, temporal autocorrelation of the data would inevitably add a significant bias to the analysis. One objective criterion for defining an event can be related to time. Assigning a certain duration can help create independent events to minimize this noise. For this protocol and the data generated with this approach, we propose creating independent 5 minute events. You can find the code for summarizing and aggregating data from a larger video-level data frame in https://github.com/Cyanopica/Ecological-interactionscamtrap-protocol/tree/main/Postprocess. The purpose of this code is to group the video data (10 s duration entries) by 5 minute intervals and then calculate summary statistics for each group calculating the sampling effort and maintaining associated data for each collapsed entry. The rartionale is that a visit sequence starts with the arrival of the animal to the plant and ends with its departure. During this visit, which typically extends beyond a 10 s duration, the animal may feed on fruits or not. Only in some instances the camera will record a fruit handling and/or ingestion, and probably even in more rare instances, will record the whole visit duration. Hence, pooling successive videos within 5 min intervals is an adequate way to record a single interaction bout. ! D. 2 AI performance metrics (image ?? - panel with example metrics for all species)! AI for image recognition is not infallible and has many limitations of how your data respond to the trained model. Thus measuring the performance of AI on your data is vital to be aware if you may be missing something important. Recognition performance can be measured in various ways and the best measure depends on the goal. Some measures used to evaluate recognition performance include precision, recall, accuracy, F-score and MCC. Precision measures the 18 proportion of correct results in the classifications, recall measures the proportion of returned positive results compared to the total true positives, accuracy measures the proportion of correct results (positive or negative), F-score combines precision and recall into an average. We provide access to the R script to help analyzing and plotting this and other performance metrics related to Image recognition. ! 19 Table 4. Some AI performance metrics used for measuring recognition performance. For a given concussion matrix TP true positive; TN true negative: FP false positive; FN false negative Measure Equation TPR True positive rate = Sensitivity TP/(TP+FN) TNR True negative rate = Specificity TN/(FP+TN) FPR False positive rate = Fall out FP/(FP+TN) FNR False negative rate = Miss rate FN/(FN+TP) PPV Positive predictive value = Precision! TP/(TP+FP) NPV Negative predictive value TN/(TN+FN) FDR False discovery rate! FP/(FP+TP) FOR False omisión rate FN/(FN+TN) ACC Accuracy (TP+TN)/(TP+TN+FP+FN) ERR Error rate (FP+FN)/(TP+TN+FP+FN) F1score Harmonic mean between TPR and PPV (2*TP)/((2*TP)+FP+FN) MCC Mathews correlation coefficient ((TP*TN)-(FP*FN))/sqrt((TP+FP)*(TP+FN)*(TN+FP)*TN+FN))