Journal article
Training recurrent networks by Evolino.
- Schmidhuber J IDSIA, 6928 Manno (Lugano), Switzerland. juergen@idsia.ch
- Wierstra D
- Gagliolo M
- Gomez F
- 2007-02-15
Published in:
- Neural computation. - 2007
Animals
Artificial Intelligence
Humans
Memory
Models, Neurological
Neural Networks, Computer
Neurons
Nonlinear Dynamics
Serial Learning
Signal Processing, Computer-Assisted
English
In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numerous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo-inverse-based linear regression. If we instead use quadratic programming to maximize the margin, we obtain the first evolutionary recurrent support vector machines. We show that Evolino-based LSTM can solve tasks that Echo State nets (Jaeger, 2004a) cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent RNNs, including gradient-based LSTM.
- Language
-
- English
- Open access status
- green
- Identifiers
-
- DOI 10.1162/neco.2007.19.3.757
- PMID 17298232
- Persistent URL
- https://sonar.ch/global/documents/139583
Statistics
Document views: 26
File downloads: