Go to main content

Project: CoastVision

Period 01. januar 2020 - 31. desember 2025
Financed by The Research Council of Norway, Oceans Program: 12 mNOK, Ministry of Trade, Industry and Fisheries: 3,15 mNOK, Ministry of Education and Research: 3,25 mNOK
Partner(s) University of Agder, University of Trento, Virginia Tech, Swedish University of Agricultural Sciences, University of California Santa Cruz, University of Plymouth
Projects is lead by Institute of Marine Research

Computer vision to expand monitoring and accelerate assessment of coastal fish (CoastVision)

A corkwing wrasse male guarding his nest
Fish in focus: A corkwing wrasse male guarding his nest while we film him with a camera rig. The black ring is an antenna that detect the fish’s unique code transmitted from a small passive integrated transponder (PIT) tag. This system allows us to collect vast amounts of ground-truth data of individual fishes in the wild.

Project summary:

It is now common to use underwater cameras to study and monitor coastal fish populations. Currently, human experts manually identify, size and count fish, frame by frame. This represents a bottleneck for upscaling deployment and data analysis.

CoastVision will apply deep learning to develop automated detection and sizing of coastal fish caught on camera. The computer vision will also be trained to identify fish in the wild by their natural “barcodes” that distinguish species, sexes and individuals, such as differences in body shape and skin coloration patterns. Individual identification and reliable re-identification is the most innovative and novel aspect of CoastVision and will open new opportunities to study behaviour, growth and survival of fish in their natural habitat.

CoastVision will focus on Atlantic cod, ballan wrasse and corkwing wrasse, all commercially important species with complex, high-contrast skin patterns. This feature will be the final step in a fully automated video analysis pipeline that will identify, track, size and count fish in video feeds from long term monitoring stations. The pipeline will be integrated into ongoing surveys and case studies whose main objective is to better understand the factors that affect the reproduction, recruitment and survival of commercially and ecologically important coastal fishes.

Further, CoastVision will support studies on short- and long-term temporal dynamics of fish communities, including detecting as the arrival of invasive species, distribution shifts and altered animal behaviour associated with climate change or other environmental stressors. Widespread adoption of camera-based monitoring with integrated computer vision will revolutionize our ability to observe, understand and respond to ecological change at scales far more refined than is currently possible.

'Mugshots” of two males of corkwing wrasse
Stable patterns: Examples of “mugshots” of two males of corkwing wrasse caught in the ongoing mark-recapture survey in Austevoll, Norway in May 2018 and again in May 2020. The unique individual facial patterns are clearly recognizable even after two years.

Key research questions: 

  1. How to solve the data-deficiency problem? A lack of labelled data of sufficient quality and quantity for training currently limits the rapid development and adoption of computer vision in marine ecology. Improved procedures and new methods for data collection is, therefore, needed. This challenge receives full attention in WP1
  2. Can we fully replace the current manual video analysis procedure of RUV/BRUV data? This requires that multiple key species are precisely identified, counted and accurately sized. This is the goal of WP2. 
  3. Is it possible to re-ID fish with high precision, and over what time scales? Our three-focal species, cod, corkwing, and ballan wrasse have complex, high-contrast skin patterns that can easily be discriminated on RUVs (Figure 1). The key question is how temporally stable these patterns are. That stability, or the lack thereof, will determine the scope of applications for visual re-ID. WP3 will develop the models to explore and answer this. 
  4.  How can computer vision improve monitoring and assessments of fish stocks? High-end computers can analyse large volumes of video much faster than manual analysis and eliminate any human error and inconsistency. With the development of re-ID as a viable alternative to tagging we can obtain key knowledge for assessments much faster and at lower cost than is currently the case. We will demonstrate how in WP4. 
Project overview
Project overview: CoastVision is divided into four interconnected work packages (WPs): WP1 – Datasets, will compile and annotate image and video data from partner projects and collect video data for re-ID with RFID. WP2 will use the data to train and validate models for Fish detection, classification and sizing. WP3 will develop fish/individual tracking and individual re-ID. WP4 - Applications, the computer vision will be deployed to process data from surveys and case studies.