MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior
Authors:
- Jennifer J. SunJSJennifer J. Sun
- Markus MarksMMMarkus Marks
- Andrew UlmerAUAndrew Ulmer
- Dipam ChakrabortyDCDipam Chakraborty
- Brian GeutherBGBrian Geuther
- Edward HayesEHEdward Hayes
- Heng JiaHJHeng Jia
- Vivek KumarVKVivek Kumar
- Sebastian OleszkoSOSebastian Oleszko
- Zachary PartridgeZPZachary Partridge
- Milan PeelmanMPMilan Peelman
- Alice RobieARAlice Robie
Preprint
Abstract
We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Stride-level analysis of mouse open field behavior using deep-learning-based pose estimationCell Reports, 2022
- Integrative Benchmarking to Advance Neurally Mechanistic Models of Human IntelligenceNeuron, 2020
- Enriching behavioral ecology with reinforcement learning methodsBehavioural Processes, 2019
- The Striatum Organizes 3D Behavior via Moment-to-Moment Action SelectionCell, 2018
- Structure of the Zebrafish Locomotor Repertoire Revealed with Unsupervised Behavioral ClusteringCurrent Biology, 2018
- Mapping the Neural Substrates of BehaviorCell, 2017
- Mapping Sub-Second Structure in Mouse BehaviorNeuron, 2015
- Toward a Science of Computational EthologyNeuron, 2014
- Drosophila: Genetics meets behaviourNature Reviews Genetics, 2001
- Isolation of a putative phospholipase c gene of drosophila, norpA, and its role in phototransductionCell, 1988