AMANet: Adaptive Multi-Path Aggregation for Learning Human 2D-3D Correspondences
- 14 December 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 979-992
Abstract
Learning human 2D-3D correspondences aims to map all human 2D pixels to a 3D human template, namely human densepose estimation, involving surface patch recognition (i.e., Index-to-Patch (I)) and regression of patch-specific UV coordinates. Despite recent progress, it remains challenging especially under the condition of in the wild, where RGB images capture real-world scenes with backgrounds, occlusions, scale variations, and postural diversity. In this paper, we address three vital problems in this task: 1) how to perceive multi-scale visual information for instances ‘`in the wild’'; 2) how to design learning objectives to address the precise instance representation harassed by multiple instances in one bounding box phenomenon; and 3) how to boost the performance of index-to-patch prediction faced by limited supervision. To tackle problems above, we propose an end-to-end deep Adaptive Multi-path Aggregation network (AMA-net) for Human DensePose Estimation. First, we introduce an adaptive multi-path aggregation algorithm to extract varying-sized instance-level feature maps, which capture multi-scale information of a bounding-box and are then utilized for analyzing different instances. Second, we adopt an instance augmentation learning objective to further distinguish the target instance from other interference instances. Third, taking advantage of 2D human parsers that are trained from sufficient annotations, we introduce a task transformer that bridges the ‘`gap’' between 2D human parsing and densepose estimation, thus benefiting the performance of densepose estimator. Experimental results on the challenging Dense-COCO dataset demonstrate that our approach sets a new record, and it significantly outperforms the state-of-the-art methods. Codes and models are publicly available.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0102200)
- National Natural Science Foundation of China (61772116, 61872064, 62020106008)
- Sichuan Science and Technology Program (2019JDTD0005)
- Tianshu AI Platform, Zhejiang Laboratory
This publication has 52 references indexed in Scilit:
- Robust Joint Graph Sparse Coding for Unsupervised Spectral Feature SelectionIEEE Transactions on Neural Networks and Learning Systems, 2016
- Fast R-CNN2015
- Geodesic Convolutional Neural Networks on Riemannian Manifolds2015
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- SMPLACM Transactions on Graphics, 2015
- FaceNet: A unified embedding for face recognition and clustering2015
- Hypercolumns for object segmentation and fine-grained localization2015
- Hashing on Nonlinear ManifoldsIEEE Transactions on Image Processing, 2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989