Look and Think Twice: Capturing Top-Down Visual Attention with Feedback Convolutional Neural Networks

Authors:
  • CCChunshui Cao
    Chunshui Cao
  • XLXianming Liu
    Xianming Liu
  • YYYi Yang
    Yi Yang
  • YYYinan Yu
    Yinan Yu
  • JWJiang Wang
    Jiang Wang
  • ZWZilei Wang
    Zilei Wang
  • YHYongzhen Huang
    Yongzhen Huang
  • LWLiang Wang
    Liang Wang
  • CHChang Huang
    Chang Huang
  • WXWei Xu
    Wei Xu
  • DRDeva Ramanan
    Deva Ramanan
  • THThomas S. Huang
    Thomas S. Huang
Top Cited Papers
Publisher website
Google Scholar
Abstract
While feedforward deep convolutional neural networks (CNNs) have been a great success in computer vision, it is important to note that the human visual cortex generally contains more feedback than feedforward connections. In this paper, we will briefly introduce the background of feedbacks in the human visual cortex, which motivates us to develop a computational feedback mechanism in deep neural networks. In addition to the feedforward inference in traditional neural networks, a feedback loop is introduced to infer the activation status of hidden layer neurons according to the "goal" of the network, e.g., high-level semantic labels. We analogize this mechanism as "Look and Think Twice." The feedback networks help better visualize and understand how deep neural networks work, and capture visual attention on expected objects, even in images with cluttered background and multiple objects. Experiments on ImageNet dataset demonstrate its effectiveness in solving tasks such as image classification and object localization.