Paper Info
Title
GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices Based on Fine-Grained Structured Weight Sparsity
Abstract
It is appealing but challenging to achieve real-time deep neural network (DNN) inference on mobile devices, because even the powerful modern mobile devices are considered as “resource-constrained” when executing large-scale DNNs. It necessitates the sparse model inference via weight pruning, i.e., DNN weight sparsity, and it is desirable to design a new DNN weight sparsity scheme that can facilitate real-time inference on mobile devices while preserving a high sparse model accuracy. This paper designs a novel mobile inference acceleration framework GRIM that is General to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and that achieves Real-time execution and high accuracy, leveraging fine-grained structured sparse model Inference and compiler optimizations for Mobiles. We start by proposing a new fine-grained structured sparsity scheme through the Block-based Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity, our GRIM framework consists of two parts: (a) the compiler optimization and code generation for real-time mobile inference; and (b) the BCR pruning optimizations for determining pruning hyperparameters and performing weight pruning. We compare GRIM with Alibaba MNN, TVM, TensorFlow-Lite, a sparse implementation based on CSR, PatDNN, and ESE (a representative FPGA inference acceleration framework for RNNs), and achieve up to <inline-formula><tex-math notation="LaTeX">$14.08\times$</tex-math></inline-formula> speedup.
Year
DOI
Venue
2022
10.1109/TPAMI.2021.3089687
IEEE Transactions on Pattern Analysis and Machine Intelligence
Keywords
DocType
Volume
Algorithms,Computers, Handheld,Deep Learning,Neural Networks, Computer
Journal
44
Issue
ISSN
Citations 
10
0162-8828
0
PageRank 
References 
Authors
0.34
22
9
Name
Order
Citations
PageRank
Wei Niu12411.21
Zhengang Li2157.27
Xiaolong Ma393.46
Dong Peiyan443.12
Gang Zhou52597176.60
Xuehai Qian632027.71
Xue Lin701.69
Yanzhi Wang81082136.11
Bin Ren98218.03