Name
Abstract | ||
|---|---|---|
Recently, maxout networks have brought significant improvements to various speech recognition and computer vision tasks. In this paper we introduce two new types of generalized maxout units, which we call p-norm and soft-maxout. We investigate their performance in Large Vocabulary Continuous Speech Recognition (LVCSR) tasks in various languages with 10 hours and 60 hours of data, and find that the p-norm generalization of maxout consistently performs well. Because, in our training setup, we sometimes see instability during training when training unbounded-output nonlinearities such as these, we also present a method to control that instability. This is the “normalization layer”, which is a nonlinearity that scales down all dimensions of its input in order to stop the average squared output from exceeding one. The performance of our proposed nonlinearities are compared with maxout, rectified linear units (ReLU), tanh units, and also with a discriminatively trained SGMM/HMM system, and our p-norm units with p equal to 2 are found to perform best. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/ICASSP.2014.6853589 | ICASSP |
Keywords | Field | DocType |
p-norm generalization,unbounded-output nonlinearities,deep neural network acoustic models,speech recognition,large vocabulary continuous speech recognition,acoustic modeling,maxout networks,deep learning,lvcsr task,normalization layer,rectified linear units,p-norm units,soft-maxout,relu,generalized maxout networks,generalisation (artificial intelligence),computer vision task,neural nets,speech processing,acoustics,training data,neural networks,speech | Rectifier (neural networks),Nonlinear system,Normalization (statistics),Square (algebra),Pattern recognition,Computer science,Speech recognition,Artificial intelligence,Artificial neural network,Hidden Markov model,Vocabulary | Conference |
ISSN | Citations | PageRank |
1520-6149 | 110 | 4.83 |
References | Authors | |
17 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Xiaohui Zhang | 1 | 194 | 19.81 |
| Jan Trmal | 2 | 235 | 20.91 |
| Daniel Povey | 3 | 2442 | 231.75 |
| Sanjeev Khudanpur | 4 | 2155 | 202.00 |