Soon it will become easier for researchers to analyse the video recordings from Mareano cruises: artificial intelligence and machine learning will be there to help them.
“When we return from a research cruise, one of the things we come back with is hours of video recordings, which each cover a 200-metre sector”, explains researcher Heidi Kristina Meyer. She has been responsible for reviewing the images to be used to train the artificial intelligence.
Time-consuming video analysis
After a cruise, the videos are analysed, with every single animal that appears being written down – and this is where computers and AI may play an important role in the future.
For now, the work is done the old-fashioned way.
“On a cruise, there are generally between 50 and 150 video stations. Over the period from 2006 to 2023, we have studied as many as 3,482 video stations, so there is a rich image library to work with”, explains Meyer.
In the long term, the researchers hope that AI will help them to do some of this work.
Some species are found in large numbers at the bottom of the sea, as can be seen from this screenshot. In such cases, using artificial intelligence will be quick and efficient, as well as giving researchers more time to analyse species that either have not been previously identified or are so difficult to identify that the process cannot be done automatically.
Teaching the computer to recognise different animals
Before the computer can help the scientists, it must learn how to do its job.
“In order to ‘teach’ the algorithm, or AI, to do what you want, it must be given access to the necessary information – and learn to use it in the right way. In this case, we want it to document which animals are at the bottom of the sea”, Meyer explains.
For this method, or model, to work, you need pictures of the species that are to be recognised. Not just one, but from all directions.
“A species can look very different on two separate pictures. Therefore, the computer needs access to a large number of pictures of each species in order to do its job”, says Meyer.
“If we use the wrong selection of images, there is a risk of false or imprecise identifications”, she says.
This is what one species of Porifera looks like when scientists take a close-up of it.
Photo:
Mareano / IMR
The images used for species recognition are more likely to look like this: an overview picture with a variety of species dotted around. Can you recognise the species from the previous picture? (See next picture for the right answer.)
Photo:
Mareano / IMR
Here, the three examples of this Porifera species are marked. This is typical of what these images look like when you don’t zoom in on the individual species.
Photo:
Mareano / IMR
A close-up of a deep-water soft coral might look like this.
Photo:
Mareano / IMR
The images used for species recognition generally look like this: an overview picture with a variety of species dotted around. Can you spot the deep-water soft coral? (See next picture for the right answer.)
Photo:
Mareano / IMR
Here the deep-water soft coral is marked. This image has been slightly colour corrected.
Photo:
Mareano / IMR
1/6
Using previously analysed images to train the computer
The road to using machine learning and artificial intelligence has been a long one, and there is still a long way to go. So far, Meyer has retrieved over a million images from the Mareano database.
“All of these have been analysed previously and include species names”, she says.
Now these images have been reviewed yet again.
“It is important for the information to be entered in the same way on all of the images. That simplifies the subsequent machine learning process”, explains Meyer. To complicate matters even further: it is not the animal itself the artificial intelligence needs to recognise, but rather its shape.
“On each image, we create a kind of frame or border around all of the animals. The AI learns to assemble them in such a way that it can use these frames as a basis for species identification”, explains Nils Piechaud. He is responsible for developing the models used by the AI in this project.
Currently they are busy establishing the borders and then teaching the AI to recognise them.
“In total, approximately 780 species are represented on the selected images. Some of the species have over 30,000 images associated with them”, says Meyer.
Now the challenge is to get the models and the AI to work as intended by the scientists.
“A human can look at a starfish once and then recognise that species. That is not the case with AI. It needs many different images from different angles and directions in order to be able to recognise a starfish, for example”, says Piechaud.
Normally the AI performs badly to start with, but as with so many other things, practice makes perfect.
“With a little bit of help from us, the model – using AI – can train itself to become better and better”, says Piechaud.
He is currently investigating exactly how much information is needed for the computer to be able to recognise the various species.
An additional challenge is that as more and more species are included, the demands on what the model must learn to do also increase – it has to keep track of both more species and more different shapes.
“Finally, once the model has learned enough, it manages to recognise the outline of the selected creatures when they appear on new videos”, he says.
Artificial intelligence at work
Many videos of the bottom of the sea need to be analysed after a research cruise. They might look like the video shown above. “Suddenly” species appear which need to be documented – that is when you see evidence of the artificial intelligence at work.
When the artificial intelligence reviews the videos, it looks specifically for the species that it has been asked to find. In this case, a sea cucumber is the target. As soon as it appears in a frame, it is allocated a unique ID number (yellow frame). The green line shows the direction in which the species is “moving”; this is how the researchers ensure that the program does not confuse the “selected” animal with other animals that may appear in the same frame. The animal is only counted once it has passed the blue line at the bottom of the screen.
The code used for this program has been developed by Nils Piechaud / the Institute of Marine Research, inspired by a similar solution developed by Ultralytics with open source code.
Cannot replace people
For the moment, the two researchers doubt that the computer can completely take over the identification process any time soon.
“Although it quickly and confidently identifies the species that it has learned, for the moment it is unlikely that it will manage to count and identify all of the species as well as a human can do”, says Piechaud.
In practice, that means a combination of AI and good old-fashioned identification work will be used for a long time to come.
“The ultimate goal is for the AI to manage the identification process itself, but for now there will be a combination of AI doing a lot of the routine work while researchers quality assure its work and also identify the more difficult or unusual species”, he says.
