Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps
- 1 July 2012
- journal article
- Published by MIT Press in Neural Computation
- Vol. 24 (7) , 1822-1852
Abstract
A new class of state-space models, reservoir models, with a fixed state transition structure (the “reservoir”) and an adaptable readout from the state space, has recently emerged as a way for time series processing and modeling. Echo state network (ESN) is one of the simplest, yet powerful, reservoir models. ESN models are generally constructed in a randomized manner. In our previous study (Rodan & Tiňo, 2011 ), we showed that a very simple, cyclic, deterministically generated reservoir can yield performance competitive with standard ESN. In this contribution, we extend our previous study in three aspects. First, we introduce a novel simple deterministic reservoir model, cycle reservoir with jumps (CRJ), with highly constrained weight values, that has superior performance to standard ESN on a variety of temporal tasks of different origin and characteristics. Second, we elaborate on the possible link between reservoir characterizations, such as eigenvalue distribution of the reservoir matrix or pseudo-Lyapunov exponent of the input-driven reservoir dynamics, and the model performance. It has been suggested that a uniform coverage of the unit disk by such eigenvalues can lead to superior model performance. We show that despite highly constrained eigenvalue distribution, CRJ consistently outperforms ESN (which has much more uniform eigenvalue coverage of the unit disk). Also, unlike in the case of ESN, pseudo-Lyapunov exponents of the selected optimal CRJ models are consistently negative. Third, we present a new framework for determining the short-term memory capacity of linear reservoir models to a high degree of precision. Using the framework, we study the effect of shortcut connections in the CRJ reservoir topology on its memory capacity.Keywords
This publication has 35 references indexed in Scilit:
- Connectivity, Dynamics, and Memory in Reservoir Computing with Binary and Analog NeuronsNeural Computation, 2010
- Pruning and regularization in reservoir computingNeurocomputing, 2009
- Liquid state machines and cultured cortical networks: The separation propertyBioSystems, 2009
- Collective Behavior of a Small-World Recurrent Neural System With Scale-Free DistributionIEEE Transactions on Neural Networks, 2007
- Feed-forward echo state networks2006
- Modeling reward functions for incomplete state representations via echo state networks2005
- Short Term Memory and Pattern Matching with Simple Echo State Networks2005
- Real-Time Computation at the Edge of Chaos in Recurrent Neural NetworksNeural Computation, 2004
- Perspectives of the high‐dimensional dynamics of neural microcircuits from the point of view of low‐dimensional readoutsComplexity, 2003
- New results on recurrent network training: unifying the algorithms and accelerating convergenceIEEE Transactions on Neural Networks, 2000