Capturing Deep Correlations with 2-Way Nets
Authors:
- Aviv EisenschtatAEAviv Eisenschtat
- Lior WolfLWLior Wolf
Preprint
Abstract
We present a noval, bi-directional mapping neural network architecture for the task of matching vectors from two data-sources. Our approach employs two tied neural network channels to project two views into a common, maximally correlated space, using the euclidean loss. To achieve both maximally correlated projections we built an encoder-decoder framework composed of two parallel networks each captures the features of each of the views. We show a direct link between the correlation loss and euclidean loss enabling the use of euclidean loss for optimizing correlation maximization problem. To overcome common euclidean regression optimization problems, we incorporated batch-normalization layers and dropout layers adapted to the model at hand. We show state of the art results on a number of computer vision matching tasks including MNIST image matching and sentence-image matching on the flickr8k and flickr30k datasets.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- RNN Fisher Vectors for Action Recognition and Image Annotation2016
- Identity Mappings in Deep Residual Networks2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Live Repetition Counting2015
- Multimodal Convolutional Neural Networks for Matching Image and Sentence2015
- Cross-scene crowd counting via deep convolutional neural networks2015
- Deep correlation for matching images and text2015
- Associating neural word embeddings with deep image representations using Fisher Vectors2015
- Nonlinear Feature Extraction Using Generalized Canonical Correlation Analysis2001
- Canonical ridge and econometrics of joint productionJournal of Econometrics, 1976