Component Selection for Craft Assembly Tasks
- 8 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (9) , 8122-8129
Abstract
Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.Keywords
All Related Versions
Funding Information
- New Energy and Industrial Technology Development Organization (JPNP20006)
This publication has 20 references indexed in Scilit:
- Discovering 3D Parts from Image Collections2021
- PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes2020
- Learning Unsupervised Hierarchical Part Decomposition of 3D Objects From a Single RGB Image2020
- Objects365: A Large-Scale, High-Quality Dataset for Object Detection2019
- PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding2019
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images2018
- Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling2018
- Im2Struct: Recovering 3D Shape Structure from a Single RGB Image2018
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction2016
- Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views2015