Name
Abstract | ||
|---|---|---|
Recurrent neural network language models (RNNLMs) have been proved superior to many other competitive language modeling techniques in terms of perplexity and word error rate. The remaining problem is the great computational complexity of RNNLMs in the output layer, resulting in long time for evaluation. Typically, a class-based RNNLM with the output layer factorized was proposed for speedup, which was still not fast enough for real-time systems. In this paper, a novel variance regularization algorithm is proposed for RNNLMs to address this problem. All the softmax-normalizing factors in the output layers are penalized to make them converge to one during the training phase, so that the output probability can be estimated efficiently via one dot-product of vectors in the output layer. The computational complexity of the output layer is reduced significantly from O(|V|H) to O(H). We further use this model for rescoring in an advanced CD-HMM-DNN system. Experimental results show that our proposed variance regularization algorithm works quite well, and the word prediction of the model is about 300 times faster than that of RNNLM without any obvious deteriorations in word error rate. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/ICASSP.2014.6854532 | ICASSP |
Keywords | Field | DocType |
recurrent neural network language models,class-based rnnlm,recurrent neural network language model,speech recognition,novel variance regularization algorithm,output probability,one dot-product,variance regularization,computational complexity,advanced cd-hmm-dnn system,competitive language modeling techniques,recurrent neural nets,softmax-normalizing factors,probability,testing,computational modeling,recurrent neural networks | Perplexity,Recurrent neural network language models,Pattern recognition,Computer science,Word error rate,Speech recognition,Time delay neural network,Regularization (mathematics),Artificial intelligence,Language model,Computational complexity theory,Speedup | Conference |
ISSN | Citations | PageRank |
1520-6149 | 8 | 0.55 |
References | Authors | |
11 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Yongzhe Shi | 1 | 47 | 5.09 |
| Wei-Qiang Zhang | 2 | 136 | 31.22 |
| Meng Cai | 3 | 68 | 8.24 |
| Jia Liu | 4 | 277 | 50.34 |