ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10588-10598
Abstract
Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rather unrealistic. ObjectFolder 1.0 is a recent dataset that introduces 100 virtualized objects with visual, acoustic, and tactile sensory data. However, the dataset is small in scale and the multisensory data is of limited quality, hampering generalization to real-world scenarios. We present ObjectFolder 2.0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1.0 in three aspects. First, our dataset is 10 times larger in the amount of objects and orders of magnitude faster in rendering time. Second, we significantly improve the multisensory rendering quality for all three modalities. Third, we show that models learned from virtual objects in our dataset successfully transfer to their real-world counterparts in three challenging tasks: object scale estimation, contact localization, and shape reconstruction. ObjectFolder 2.0 offers a new path and testbed for multisensory learning in computer vision and robotics. The dataset is available at https://github.com/rhgao/ObjectFolder.Keywords
All Related Versions
Funding Information
- Toyota Research Institute (TRI)
- Samsung
- Autodesk
- Amazon
This publication has 45 references indexed in Scilit:
- Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification2016
- ObjectNet3D: A Large Scale Database for 3D Object Recognition2016
- Computational design of metallophone contact soundsACM Transactions on Graphics, 2015
- Localizing the object contact through matching tactile features with visual map2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Parsing IKEA Objects: Fine Pose Estimation2013
- Example-guided physically based modal sound synthesisACM Transactions on Graphics, 2013
- An ES-FEM for accurate analysis of 3D mid-frequency acoustics using tetrahedron meshComputers & Structures, 2012
- Sequential Monte Carlo Methods for Dynamic SystemsJournal of the American Statistical Association, 1998
- Ray tracing volume densitiesACM SIGGRAPH Computer Graphics, 1984