Deep Canonical Time Warping for Simultaneous Alignment and Representation Learning of Sequences
- 8 June 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 40 (5) , 1128-1138
Abstract
Machine learning algorithms for the analysis of time-series often depend on the assumption that utilised data are temporally aligned. Any temporal discrepancies arising in the data is certain to lead to ill-generalisable models, which in turn fail to correctly capture properties of the task at hand. The temporal alignment of time-series is thus a crucial challenge manifesting in a multitude of applications. Nevertheless, the vast majority of algorithms oriented towards temporal alignment are either applied directly on the observation space or simply utilise linear projections-thus failing to capture complex, hierarchical non-linear representations that may prove beneficial, especially when dealing with multi-modal data (e.g., visual and acoustic information). To this end, we present Deep Canonical Time Warping (DCTW), a method that automatically learns non-linear representations of multiple time-series that are (i) maximally correlated in a shared subspace, and (ii) temporally aligned. Furthermore, we extend DCTW to a supervised setting, where during training, available labels can be utilised towards enhancing the alignment process. By means of experiments on four datasets, we show that the representations learnt significantly outperform state-of-the-art methods in temporal alignment, elegantly handling scenarios with heterogeneous feature sets, such as the temporal alignment of acoustic and visual information.Keywords
Funding Information
- Department of Computing, Imperial College London
- EPSRC (EP/J017787/1 (4D-FAB))
- Fidipro Program of Tekes (1849/31/2015)
- European Community’s Horizon 2020 Framework Programme (645378)
This publication has 25 references indexed in Scilit:
- Generalized Canonical Time WarpingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Deep learning for robust feature generation in audiovisual emotion recognition2013
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006
- Style translation for human motion2005
- A linear time algorithm for computing exact Euclidean distance transforms of binary images in arbitrary dimensionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003
- Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing dataIEEE Transactions on Image Processing, 2002
- Motion signal processing1995
- X-ray microbeam speech production databaseThe Journal of the Acoustical Society of America, 1990
- On the Hidden Markov Model and Dynamic Time Warping for Speech Recognition-A Unified ViewAT&T Bell Laboratories Technical Journal, 1984
- Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentencesIEEE Transactions on Acoustics, Speech, and Signal Processing, 1980