Paper Info
Title
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy.
Abstract
Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems -- the models (often deep networks or wide networks or both) are compute and memory intensive. Low precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low precision networks can be significantly improved by using knowledge distillation techniques. We call our approach Apprentice and show state-of-the-art accuracies using ternary precision and 4-bit precision for many variants of ResNet architecture on ImageNet dataset. We study three schemes in which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.
Year
Venue
Field
2017
international conference on learning representations
Object detection,Software deployment,Computer science,Inference,Distillation,Artificial intelligence,Deep learning,Contextual image classification,Memory footprint,Model compression,Machine learning
DocType
Volume
Citations 
Journal
abs/1711.05852
20
PageRank 
References 
Authors
0.66
17
2
Name
Order
Citations
PageRank
Asit K. Mishra1121646.21
Debbie Marr217512.39