Interpretable Structure-Evolving LSTM
Authors:
- Xiaodan LiangXLXiaodan Liang
- Liang LinLLLiang Lin
- Xiaohui ShenXSXiaohui Shen
- Jiashi FengJFJiashi Feng
- Shuicheng YanSYShuicheng Yan
- Eric P. XingEXEric P. Xing
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 2175-2184
Abstract
This paper develops a general framework for learning interpretable data representation via Long Short-Term Memory (LSTM) recurrent neural networks over hierarchal graph structures. Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization. We thus call this model the structure-evolving LSTM. In particular, starting with an initial element-level graph representation where each node is a small data element, the structure-evolving LSTM gradually evolves the multi-level graph representations by stochastically merging the graph nodes with high compatibilities along the stacked LSTM layers. In each LSTM layer, we estimate the compatibility of two connected nodes from their corresponding LSTM gate outputs, which is used to generate a merging probability. The candidate graph structures are accordingly generated where the nodes are grouped into cliques with their merging probabilities. We then produce the new graph structure with a Metropolis-Hasting algorithm, which alleviates the risk of getting stuck in local optimums by stochastic sampling with an acceptance probability. Once a graph structure is accepted, a higher-level graph is then constructed by taking the partitioned cliques as its nodes. During the evolving process, representation becomes more abstracted in higher-levels where redundant information is filtered out, allowing more efficient propagation of long-range data dependencies. We evaluate the effectiveness of structure-evolving LSTM in the application of semantic object parsing and demonstrate its advantage over state-of-the-art LSTM models on standard benchmarks.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Matching-CNN meets KNN: Quasi-parametric human parsing2015
- Scene labeling with LSTM recurrent neural networks2015
- Deep Human Parsing with Active Template RegressionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks2015
- Caffe2014
- Texture Classification Using 2D LSTM Networks2014
- Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts2014
- Paper Doll Parsing: Retrieving Similar Styles to Parse Clothing Items2013
- Speech recognition with deep recurrent neural networks2013
- Image segmentation by data-driven markov chain monte carloIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002