A Multi-Modal, Discriminative and Spatially Invariant CNN for RGB-D Object Labeling
- 30 August 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 40 (9) , 2051-2065
Abstract
While deep convolutional neural networks have shown a remarkable success in image classification, the problems of inter-class similarities, intra-class variances, the effective combination of multi-modal data, and the spatial variability in images of objects remain to be major challenges. To address these problems, this paper proposes a novel framework to learn a discriminative and spatially invariant classification model for object and indoor scene recognition using multi-modal RGB-D imagery. This is achieved through three postulates: 1) spatial invariance-this is achieved by combining a spatial transformer network with a deep convolutional neural network to learn features which are invariant to spatial translations, rotations, and scale changes, 2) high discriminative capability-this is achieved by introducing Fisher encoding within the CNN architecture to learn features which have small inter-class similarities and large intra-class compactness, and 3) multi-modal hierarchical fusion-this is achieved through the regularization of semantic segmentation to a multi-modal CNN architecture, where class probabilities are estimated at different hierarchical levels (i.e., imageand pixel-levels), and fused into a Conditional Random Field (CRF)-based inference hypothesis, the optimization of which produces consistent class labels in RGB-D images. Extensive experimental evaluations on RGB-D object and scene datasets, and live video streams (acquired from Kinect) show that our framework produces superior object and scene classification results compared to the state-of-the-art methods.Keywords
Funding Information
- Australian Research Council (DP150100294, DE120102960)
This publication has 32 references indexed in Scilit:
- Multimodal deep learning for robust RGB-D object recognition2015
- RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features2015
- Efficient RGB-D object categorization using cascaded ensembles of randomized decision trees2015
- Unsupervised feature learning for 3D scene labeling2014
- From Low-Cost Depth Sensors to CAD: Cross-Domain 3D Shape Retrieval via Regression Tree Fields2014
- Selective Search for Object RecognitionInternational Journal of Computer Vision, 2013
- Unsupervised Feature Learning for RGB-D Based Object Recognition2013
- A learned feature descriptor for object recognition in RGB-D data2012
- Depth kernel descriptors for object recognition2011
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001