Modality and Component Aware Feature Fusion for RGB-D Scene Classification
Authors:
- Anran WangAWAnran Wang
- Jianfei CaiJCJianfei Cai
- Jiwen LuJLJiwen Lu
- Tat-Jen ChamTCTat-Jen Cham
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5995-6004
Abstract
While convolutional neural networks (CNN) have been excellent for object recognition, the greater spatial variability in scene images typically meant that the standard full-image CNN features are suboptimal for scene classification. In this paper, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV) encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation comprising multiple modalities of RGB, HHA and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity - that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal non-sparsity - within these discriminative components, all modalities have important contribution. In our framework, these are implemented through regularization terms applying group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we were able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2.Keywords
This publication has 16 references indexed in Scilit:
- Second-order constrained parametric proposals and sequential search-based structured prediction for semantic segmentation in RGB-D images2015
- SUN RGB-D: A RGB-D scene understanding benchmark suite2015
- Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic SegmentationInternational Journal of Computer Vision, 2014
- SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels2013
- Image Classification with the Fisher Vector: Theory and PracticeInternational Journal of Computer Vision, 2013
- Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images2013
- A Category-Level 3D Object Dataset: Putting the Kinect to Work2013
- Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learningBioinformatics, 2012
- Human action recognition by learning bases of action attributes and parts2011
- Model Selection and Estimation in Regression with Grouped VariablesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005