CRISP: Object Pose and Shape Estimation with Test-Time Adaptation
Preprint
Abstract
We consider the problem of estimating object pose and shape from an RGB-D image. Our first contribution is to introduce CRISP, a category-agnostic object pose and shape estimation pipeline. The pipeline implements an encoder-decoder model for shape estimation. It uses FiLM-conditioning for implicit shape reconstruction and a DPT-based network for estimating pose-normalized points for pose estimation. As a second contribution, we propose an optimization-based pose and shape corrector that can correct estimation errors caused by a domain gap. Observing that the shape decoder is well behaved in the convex hull of known shapes, we approximate the shape decoder with an active shape model, and show that this reduces the shape correction problem to a constrained linear least squares problem, which can be solved efficiently by an interior point algorithm. Third, we introduce a self-training pipeline to perform self-supervised domain adaptation of CRISP. The self-training is based on a correct-and-certify approach, which leverages the corrector to generate pseudo-labels at test time, and uses them to self-train CRISP. We demonstrate CRISP (and the self-training) on YCBV, SPE3R, and NOCS datasets. CRISP shows high performance on all the datasets. Moreover, our self-training is capable of bridging a large domain gap. Finally, CRISP also shows an ability to generalize to unseen objects. Code and pre-trained models will be available on https://web.mit.edu/sparklab/research/crisp_object_pose_shape/.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision2025
- A Correct-and-Certify Approach to Self-Supervise Object Pose Estimators via Ensemble Self-Training2023
- Certifiable Object Pose Estimation: Foundations, Learning Models, and Self-TrainingIEEE Transactions on Robotics, 2023
- Category-Level Metric Scale Object Shape and Pose EstimationIEEE Robotics and Automation Letters, 2021
- The State of Industrial Robotics: Emerging Technologies, Challenges, and Key Research DirectionsFoundations and Trends in Robotics, 2021
- Least-Squares Fitting of Two 3-D Point SetsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1987