Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates
Preprint
Abstract
Objective: Neural decoders for dexterous finger movements are limited by the accuracy with which ground truth movement is measured. In non-human primates (NHPs), hand kinematics are typically recorded with instrumentation that constrains movement dimensionality. We evaluated whether a markerless camera system could replace conventional methods for tracking movements in real time. Approach: We developed Freebie, a real-time markerless system for tracking NHP finger movements. Keypoints detected online were used in place of manipulandum bend sensor signals. Two NHPs performed tasks by flexing fingers to move a virtual hand into target zones. The virtual hand was directly controlled using bend sensors or Freebie-derived kinematics, or indirectly controlled through a Kalman filter (KF) decoding kinematics from forearm electromyographic (EMG) activity. We quantified Freebie’s accuracy relative to manual labels, agreement with bend sensor measurements, and latencies. We assessed online performance under Freebie control and compared EMG KFs trained using Freebie-derived labels with those trained using bend-sensor-derived labels. Main Results: We observed a median end-to-end latency of 15.8 ms at a 30 FPS camera acquisition rate, and 16.2 ms at 60 FPS. Freebie labels closely tracked bend sensor positions (median |r|=0.96). Median keypoint errors were 1.7 to 6.5 pixels, below the 20-pixel width of the smallest tracked finger. Direct Freebie control did not differ from bend sensor control on five of six online metrics; path efficiency was higher with Freebie. EMG KFs trained on Freebie-derived labels performed similarly to those trained on bend sensor labels online. When evaluated offline, models trained with Freebie had lower error. Significance: These findings indicate that bend sensors are not required to obtain finger labels for training a decoder: markerless keypoints alone provided effective labels. This is an important first step towards real-time tracking of more complex movements, beyond what could be easily measured with a manipulandum.Funding Information
- National Institutes of Health (R01NS134834)
- National Institutes of Health (R01DA063389)
- National Institute of General Medical Sciences (T32GM141746)
- Italian Ministry of University and Research (PE0000006)
- U.S. National Science Foundation (2223822)
- U.S. National Science Foundation (1926576)
- U.S. National Science Foundation (DGE-2241144)
- Alfred P. Sloan Foundation (FG-2024-21442)
- Burroughs Wellcome Fund (1021865.01)
This publication has 22 references indexed in Scilit:
- A software platform for real-time and adaptive neuroscience experimentsNature Communications, 2025
- A high-performance brain–computer interface for finger decoding and quadcopter game control in an individual with paralysisNature Medicine, 2025
- Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source toolsNature Methods, 2024
- Neurotechnologies to restore hand functionsNature Reviews Bioengineering, 2023
- Real-time brain-machine interface in non-human primates achieves high-velocity prosthetic finger movements using a shallow feedforward neural network decoderNature Communications, 2022
- Real-time linear prediction of simultaneous and independent movements of two finger groups using an intracortical brain-machine interfaceNeuron, 2021
- Anipose: A toolkit for robust markerless 3D pose estimationCell Reports, 2021
- Geometric deep learning enables 3D kinematic profiling across species and environmentsNature Methods, 2021
- Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudioNature Communications, 2020
- DeepLabCut: markerless pose estimation of user-defined body parts with deep learningNature Neuroscience, 2018