Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders
Authors:
- Matthew R WhitewayMWMatthew R Whiteway
- Dan BidermanDBDan Biderman
- Mario DipoppaMDMario Dipoppa
- E. Kelly BuchananEBE. Kelly Buchanan
- Niccolò BonacchiNBNiccolò Bonacchi
- Nathaniel J MiskaNMNathaniel J Miska
- Jean-Paul NoelJNJean-Paul Noel
- Erica RodriguezERErica Rodriguez
- Michael SchartnerMSMichael Schartner
Preprint
Abstract
Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.Keywords
All Related Versions
- Published version: PLOS Computational Biology, 17 (9), e1009439.
This publication has 23 references indexed in Scilit:
- A size principle for leg motor control in DrosophilabioRxiv, 2019
- DeepBehavior: A Deep Learning Toolbox for Automated Analysis of Animal and Human Behavior Imaging DataFrontiers in Systems Neuroscience, 2019
- Spontaneous behaviors drive multidimensional, brainwide activityScience, 2019
- Structure of the Zebrafish Locomotor Repertoire Revealed with Unsupervised Behavioral ClusteringCurrent Biology, 2018
- Semi Supervised Autoencoder2016
- Attribute2Image: Conditional Image Generation from Visual Attributes2016
- Suite2p: beyond 10,000 neurons with standard two-photon microscopybioRxiv, 2016
- Demixed principal component analysis of neural population dataeLife, 2016
- Deep extreme learning machines: supervised autoencoding architecture for classificationNeurocomputing, 2016
- Mapping Sub-Second Structure in Mouse BehaviorNeuron, 2015