Name
Abstract | ||
|---|---|---|
Recently, Maxout networks have demonstrated state-of-the-art performance on several machine learning tasks, which has fueled aggressive research on Maxout networks and generalizations thereof. In this work, we propose the utilization of order statistics as a generalization of the max non-linearity. A particularly general example of an order-statistic non-linearity is the “sortout” non-linearity, which outputs all input activations, but in sorted order. Such Order-statistic networks (OSNs), in contrast with other recently proposed generalizations of Maxout networks, leave the determination of the interpolation weights on the activations to the network, and remain conditionally linear given the input, and so are well suited for powerful model aggregation techniques such as dropout, drop connect, and annealed dropout. Experimental results demonstrate that the use of order statistics rather than Maxout networks can lead to substantial improvements in the word error rate (WER) performance of automatic speech recognition systems. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/SLT.2014.7078561 | Spoken Language Technology Workshop |
Keywords | Field | DocType |
multilayer perceptrons,speech recognition,statistical analysis,OSN,WER performance improvement,annealed dropout,automatic speech recognition systems,deep-order statistic networks,drop connect,dropout,input activations,interpolation weights,linear network,max-nonlinearity,maxout networks,model aggregation techniques,network activation,order-statistic nonlinearity,sortout nonlinearity,word error rate performance improvement,Deep Neural Networks,Maxout Networks,Multi-Layer Perceptrons,Order Statistic Networks,Rectified Linear Units | Rectifier (neural networks),Pattern recognition,Computer science,Generalization,Word error rate,Interpolation,Speech recognition,Artificial intelligence,Order statistic,Model aggregation,Deep neural networks | Conference |
ISSN | Citations | PageRank |
2639-5479 | 8 | 0.63 |
References | Authors | |
12 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Steven J. Rennie | 1 | 302 | 24.77 |
| Vaibhava Goel | 2 | 376 | 41.25 |
| Samuel Thomas | 3 | 536 | 46.88 |