BIOCLIP: A Vision Foundation Model for the Tree of Life
Authors:
Preprint
Abstract
Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion of computational methods and tools, particularly computer vision, for extracting biologically relevant information from images for science and conservation. Yet most of these are bespoke approaches designed for a specific task and are not easily adaptable or extendable to new questions, contexts, and datasets. A vision model for general organismal biology questions on images is of timely need. To approach this, we curate and release TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images. We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge. We rigorously benchmark our approach on diverse fine-grained biology classification tasks, and find that BioCLIP consistently and substantially outperforms existing baselines (by 17% to 20% absolute). Intrinsic evaluation reveals that BioCLIP has learned a hierarchical representation conforming to the tree of life, shedding light on its strong generalizability. Our code, models and data will be made available at https://github.com/Imageomics/bioclip.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Combined scaling for zero-shot transfer learningNeurocomputing, 2023
- Contrastive language and vision learning of general fashion conceptsScientific Reports, 2022
- Identification of morphologically cryptic species with computer vision models: wall lizards (Squamata: Lacertidae: Podarcis) as a case studyZoological Journal of the Linnean Society, 2022
- Unsupervised machine learning reveals mimicry complexes in bumblebees occur along a perceptual continuumProceedings Of The Royal Society B-Biological Sciences, 2019
- Identification of plant leaf diseases using a nine-layer deep convolutional neural networkComputers and Electrical Engineering, 2019
- Citizen science can improve conservation science, natural resource management, and environmental protectionBiological Conservation, 2017
- Scaling‐up camera traps: monitoring the planet's biodiversity with networks of remote sensorsFrontiers in Ecology and the Environment, 2016
- Synthesis of phylogeny and taxonomy into a comprehensive tree of lifeProceedings of the National Academy of Sciences, 2015
- Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savannaScientific Data, 2015
- The eBird enterprise: An integrated approach to development and application of citizen scienceBiological Conservation, 2014