Name
Abstract | ||
|---|---|---|
This paper explores asynchronous stochastic optimization for sequence training of deep neural networks. Sequence training requires more computation than frame-level training using pre-computed frame data. This leads to several complications for stochastic optimization, arising from significant asynchrony in model updates under massive parallelization, and limited data shuffling due to utterance-chunked processing. We analyze the impact of these two issues on the efficiency and performance of sequence training. In particular, we suggest a framework to formalize the reasoning about the asynchrony and present experimental results on both small and large scale Voice Search tasks to validate the effectiveness and efficiency of asynchronous stochastic optimization. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/ICASSP.2014.6854672 | Acoustics, Speech and Signal Processing |
Keywords | Field | DocType |
neural nets,speech processing,speech recognition,stochastic programming,asynchronous stochastic optimization,deep neural networks,frame-level training,large scale voice search tasks,limited data shuffling,massive parallelization,pre-computed frame data,sequence training,small scale voice search tasks,speech recognition,utterance-chunked processing,acoustic modeling,asynchronous stochastic optimization,neural networks,sequence training,speech recognition | Asynchronous communication,Stochastic optimization,Asynchrony,Computer science,Shuffling,Artificial intelligence,Machine learning,Voice search,Deep neural networks,Computation | Conference |
ISSN | Citations | PageRank |
1520-6149 | 26 | 2.25 |
References | Authors | |
7 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Georg Heigold | 1 | 539 | 37.69 |
| E. McDermott | 2 | 514 | 88.33 |
| Vincent Vanhoucke | 3 | 4735 | 213.63 |
| Andrew Senior | 4 | 4687 | 260.55 |