TossNet: Learning to Accurately Measure and Predict Robot Throwing of Arbitrary Objects in Real Time With Proprioceptive Sensing
- 18 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 40 (15523098) , 3232-3251
Abstract
Accurate measuring and modeling of dynamic robot manipulation (e.g., tossing and catching) is particularly challenging, due to the inherent nonlinearity, complexity, and uncertainty in high-speed robot motions and highly dynamic robot-object interactions happening in very short distances and times. Most studies leverage extrinsic sensors such as visual and tactile feedback towards task or object-centric modeling of manipulation dynamics, which however may hit bottleneck due to the significant cost and complexity, e.g., the environmental restrictions. In this work, we investigate whether using solely the on-board proprioceptive sensory modalities can effectively capture and characterize dynamic manipulation processes. In particular, we present an object-agnostic strategy to learn the robot toss dynamics of arbitrary unknown objects from the spatio-temporal variations of robot toss movements and wrist-force/torque (F/T) observations. We then propose TossNet, an end-to-end formulation that jointly measures the robot toss dynamics and predicts the resulting flying trajectories of the tossed objects. Experimental results in both simulation and real-world scenarios demonstrate that our methods can accurately model the robot toss dynamics of both seen and unseen objects, and predict their flying trajectories with superior prediction accuracy in nearly real-time. Ablative results are also presented to demonstrate the effectiveness of each proprioceptive modality and their correlations in modeling the toss dynamics. Case studies show that TossNet can be applied on various real robot platforms for challenging tossing-centric robot applications, such as blind juggling and high-precise robot pitching. A video of the real robot experiments can be found at https://youtu.be/y_Bz9debQyg .Keywords
This publication has 79 references indexed in Scilit:
- TidyBot: personalized robot assistance with large language modelsAutonomous Robots, 2023
- Kinematic and dynamic analysis of a dexterous multi-fingered delta robot for object catchingRobotica, 2022
- Control of robotic object pivoting based on tactile sensingMechatronics, 2021
- Physics-informed machine learningNature Reviews Physics, 2021
- Robotic Pick-and-Toss Facilitates Urban Waste Sorting2020
- Genetic Algorithm2018
- Robust proprioceptive grasping with a soft robot handAutonomous Robots, 2018
- Social LSTM: Human Trajectory Prediction in Crowded Spaces2016
- Estimating the non-linear dynamics of free-flying objectsRobotics and Autonomous Systems, 2012
- A dynamic object manipulation approach to dynamic biped locomotionRobotics and Autonomous Systems, 2008