Paper Info
Title
Efficient Knowledge Distillation From An Ensemble Of Teachers
Abstract
This paper describes the effectiveness of knowledge distillation using teacher student training for building accurate and compact neural networks. We show that with knowledge distillation. information from multiple acoustic models like very deep VGG networks and Long Short-Term Memory (LSTM) models can be used to train standard convolutional neural network (CNN) acoustic models for a variety of systems requiring a quick turnaround. We examine two strategies to leverage multiple teacher labels for training student models. In the first technique, the weights of the student model are updated by switching teacher labels at the minibatch level. In the second method. student models are trained on multiple streams of information from various teacher distributions via data augmentation. We show that standard CNN acoustic models can achieve comparable recognition accuracy with much smaller number of model parameters compared to teacher VGG and LSTM acoustic models. Additionally we also investigate the effectiveness of using broadband teacher labels as privileged knowledge for training better narrowband acoustic models within this framework. We show the benefit of this simple technique by training narrow band student models with broadband teacher soft labels on the Aurora 4 task.
Year
DOI
Venue
2017
10.21437/Interspeech.2017-614
18TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION (INTERSPEECH 2017), VOLS 1-6: SITUATED INTERACTION
Keywords
Field
DocType
Speech recognition, knowledge distillation, teacher-student, CNN, VGG, LSTM, bandwidth
Computer science,Distillation,Natural language processing,Artificial intelligence,Machine learning
Conference
ISSN
Citations 
PageRank 
2308-457X
7
0.63
References 
Authors
0
6
Name
Order
Citations
PageRank
Takashi Fukuda1104.86
Masayuki Suzuki2235.88
Gakuto Kurata310719.06
Samuel Thomas453646.88
Jia Cui5946.26
Bhuvana Ramabhadran61779153.83