Title
Preserving activations in recurrent neural networks based on surprisal.
Abstract
Learning hierarchical abstractions from sequences is a challenging and open problem for recurrent neural networks (RNNs). This is mainly due to the difficulty of detecting features that span over long time distances with also different frequencies. In this paper, we address this challenge by introducing surprisal-based activation, a novel method to preserve activations and skip updates depending on encoding-based information content. The preserved activations can be considered as temporal shortcuts with perfect memory. We present a preliminary analysis by evaluating surprisal-based activation on language modeling with the Penn Treebank corpus and find that it can improve performance when compared to baseline RNNs and Long Short-Term Memory (LSTM) networks.
Year
DOI
Venue
2019
10.1016/j.neucom.2018.11.092
Neurocomputing
Keywords
Field
DocType
Recurrent neural networks,Conditional computation,Language modeling
Open problem,Abstraction,Recurrent neural network,Speech recognition,Artificial intelligence,Treebank,Mathematics,Machine learning,Encoding (memory)
Journal
Volume
ISSN
Citations 
342
0925-2312
0
PageRank 
References 
Authors
0.34
15
3
Name
Order
Citations
PageRank
Tayfun Alpay153.15
Fares Abawi200.34
Stefan Wermter31100151.62