Deep Imputation for Skeleton Data (DISK) for Behavioral Science
Preprint
Abstract
Pose estimation methods and motion capture systems have opened doors to quantitative measurements of animal kinematics. However, these methods are not perfect and contain missing data. Our method, Deep Imputation for Skeleton data (DISK), leverages deep learning algorithms to learn dependencies between keypoints and their dynamics to impute missing tracking data. We developed an unsupervised training scheme, which does not rely on manual annotations, and tested several neural network architectures for the imputation task. We found that transformer outperforms other architectures including graph convolutional networks that were developed specifically for skeleton-based action recognition. We demonstrate the usability and performance of our imputation method on seven different animal skeletons including two multi-animal set-ups. With an optional estimated imputation error, DISK enables behavior scientists to assess the reliability of the imputed data. The imputed recordings allow to detect more episodes of motion, such as steps, and to obtain more statistically robust results when comparing these episodes between experimental conditions. While animal behavior experiments are expensive and complex, tracking errors make sometimes large portions of the experimental data unusable. DISK allows for filling in the missing information and for taking full advantage of the rich behavioral data. This stand-alone imputation package, freely available at https://github.com/bozeklab/DISK.git, is applicable to results of any tracking method (marker-based or markerless) and allows for any type of downstream analysis.Keywords
All Related Versions
- Published version: Nature Methods, 23 (1), 236.
This publication has 30 references indexed in Scilit:
- Differences between lower extremity joint running kinetics captured by marker-based and markerless systems were speed dependentJournal of Sport and Health Science, 2024
- SAITS: Self-attention-based imputation for time seriesExpert Systems with Applications, 2023
- Animal Pose Tracking: 3D Multimodal Dataset and Token-based Pose OptimizationInternational Journal of Computer Vision, 2022
- Graph transformer network with temporal kernel attention for skeleton-based action recognitionKnowledge-Based Systems, 2022
- Distinct movement patterns generate stages of spider web buildingCurrent Biology, 2021
- Anipose: A toolkit for robust markerless 3D pose estimationCell Reports, 2021
- B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviorsNature Communications, 2021
- DeepAR: Probabilistic forecasting with autoregressive recurrent networksInternational Journal of Forecasting, 2020
- Fast animal pose estimation using deep neural networksNature Methods, 2018
- Forecasting stock markets using wavelet transforms and recurrent neural networks: An integrated system based on artificial bee colony algorithmApplied Soft Computing, 2011