Zero‐shot animal behaviour classification with vision‐language foundation models
Open Access
- 20 May 2025
- journal article
- Published by Wiley in Methods in Ecology and Evolution
- Vol. 16 (7) , 1460-1472
Abstract
Understanding the behaviour of animals in their natural habitats is critical to ecology and conservation. Camera traps are a powerful tool to collect such data with minimal disturbance. They however produce a very large quantity of images, which can make human‐based annotation cumbersome or even impossible. While automated species identification with artificial intelligence has made impressive progress, the automatic classification of animal behaviours in camera trap images remains a developing field. Here, we explore the potential of foundation models, specifically vision‐language models (VLMs), to perform this task without the need to first train a model, which would require some level of human‐based annotation. Using two datasets, of alpine and African fauna, we investigate the zero‐shot capabilities of different kinds of recent VLMs to predict behaviours and estimate behaviour‐specific diel activity patterns in three ungulate species. By comparing our predictions to behaviours annotated by participatory science, our results show that using these automatic methods, it is possible to achieve F1‐score as high as 86.39% in behaviour classification and produce activity patterns that closely align with those derived from participatory science data (overlap indexes between 84% and 90%). These findings demonstrate the potential of foundation models and vision‐language models in ecological research. Ecologists are encouraged to adopt these new methods and leverage their full capabilities to facilitate ecological studies.Keywords
This publication has 22 references indexed in Scilit:
- Seasonal activity patterns and home range sizes of wolves in the human‐dominated landscape of northeast TürkiyeWildlife Biology, 2024
- Camera traps reveal seasonal variation in activity and occupancy of the Alpine mountain hare Lepus timidus varronisWildlife Biology, 2024
- Being confident in confidence scores: calibration in deep learning models for camera trap image sequencesbioRxiv, 2023
- Automated visitor and wildlife monitoring with camera traps and machine learningRemote Sensing in Ecology and Conservation, 2023
- Daily activity timing in the AnthropoceneTrends in Ecology & Evolution, 2022
- The inappropriate use of time‐to‐independence biases estimates of activity patterns of free‐ranging mammals derived from camera trapsEcology and Evolution, 2022
- Identification of animals and recognition of their actions in wildlife videos using deep learning techniquesEcological Informatics, 2021
- Automatically identifying, counting, and describing wild animals in camera-trap images with deep learningProceedings of the National Academy of Sciences, 2018
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savannaScientific Data, 2015