A promptable foundation model enables automated multi-task dataset construction and real-time pose estimation in mice

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Behavioural experiments often hinge on moments when animals crowd together. In these situations, existing pose-estimation methods usually fail because detected crops include neighbouring animals, and their body parts are easily confused between individuals. We solved it with DeepLabSAM, a pipeline that segments and tracks every animal from a single text prompt using the SAM 3 foundation model, followed by a mask-gated pretrained pose estimation network from DeepLabCut. Across recordings of multiple interacting mice, the proposed method substantially reduces misassigned keypoints under heavy occlusion. Furthermore, we used DeepLabSAM to automatically construct the Multi-Task Mouse Behaviour (MTMB) dataset, comprising 14,300 frames spanning nine behavioural assays, with each instance annotated for bounding box, mask, identity, and pose. SuperAnimal pretrained pose models transferred unpredictably across this dataset. Conversely, MouseLite, our RF-DETR pose-estimation model trained on the MTMB dataset, matched their in-distribution accuracy, generalized better to unseen assays, and achieved real-time (faster-than-acquisition) inference. Combined, DeepLabSAM, MTMB dataset, and MouseLite provide a scalable framework for robust, cross-assay behavioural pose estimation of freely interacting mice.
Funding Information
  • Centre National de la Recherche Scientifique
  • Centre National de la Recherche Scientifique (MITI 80 PRIME NEUROJAC)

This publication has 25 references indexed in Scilit: