Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data
Authors:
- Jing WangJWJing Wang
- Yu ChengYCYu Cheng
- Rogerio Schmidt FerisRFRogerio Schmidt Feris
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2295-2304
Abstract
The way people look in terms of facial attributes (ethnicity, hair color, facial hair, etc.) and the clothes or accessories they wear (sunglasses, hat, hoodies, etc.) is highly dependent on geo-location and weather condition, respectively. This work explores, for the first time, the use of this contextual information, as people with wearable cameras walk across different neighborhoods of a city, in order to learn a rich feature representation for facial attribute classification, without the costly manual annotation required by previous methods. By tracking the faces of casual walkers on more than 40 hours of egocentric video, we are able to cover tens of thousands of different identities and automatically extract nearly 5 million pairs of images connected by or from different face tracks, along with their weather and location context, under pose and lighting variations. These image pairs are then fed into a deep network that preserves similarity of images connected by the same track, in order to capture identity-related attribute features, and optimizes for location and weather prediction to capture additional facial attribute features. Finally, the network is fine-tuned with manually annotated samples. We perform an extensive experimental analysis on wearable data and two standard benchmark datasets based on web images (LFWA and CelebA). Our method outperforms by a large margin a network trained from scratch. Moreover, even without using manually annotated identity labels for pre-training as in previous methods, our approach achieves results that are better than the state of the art.Keywords
This publication has 39 references indexed in Scilit:
- Deep Learning Face Attributes in the Wild2015
- Two Birds, One Stone: Jointly Learning Binary Code for Large-Scale Face Image Retrieval and Attributes Prediction2015
- FaceNet: A unified embedding for face recognition and clustering2015
- Delving into egocentric actions2015
- First-person pose recognition using egocentric workspaces2015
- Transferring a semantic representation for person re-identification and search2015
- A Deep Sum-Product Architecture for Robust Facial Attributes Analysis2013
- Street-to-shop2012
- Image ranking and retrieval based on multi-attribute queries2011
- Deep learning from temporal coherence in video2009