Paper Info
Title
The Next 700 Accelerated Layers: From Mathematical Expressions of Network Computation Graphs to Accelerated GPU Kernels, Automatically
Abstract
Deep learning frameworks automate the deployment, distribution, synchronization, memory allocation, and hardware acceleration of models represented as graphs of computational operators. These operators wrap high-performance libraries such as cuDNN or NNPACK. When the computation does not match any predefined library call, custom operators must be implemented, often at high engineering cost and performance penalty, limiting the pace of innovation. To address this productivity gap, we propose and evaluate: (1) a domain-specific language with a tensor notation close to the mathematics of deep learning; (2) a Just-In-Time optimizing compiler based on the polyhedral framework; (3) carefully coordinated linear optimization and evolutionary algorithms to synthesize high-performance CUDA kernels; (4) the transparent integration of our flow into PyTorch and Caffe2, providing the fully automatic synthesis of high-performance GPU kernels from simple tensor algebra. The performance is comparable to, and often exceeds the performance of, highly tuned libraries.
Year
DOI
Venue
2020
10.1145/3355606
ACM Transactions on Architecture and Code Optimization (TACO)
Keywords
Field
DocType
Deep learning layers, GPU acceleration, polyhedral compilation
Evolutionary algorithm,Computer science,CUDA,Parallel computing,Optimizing compiler,Memory management,Operator (computer programming),Artificial intelligence,Hardware acceleration,Ricci calculus,Deep learning
Journal
Volume
Issue
ISSN
16
4
1544-3566
Citations 
PageRank 
References 
2
0.36
0
Authors
9
Name
Order
Citations
PageRank
Nicolas Vasilache135419.45
Oleksandr Zinenko2162.61
Theodoros Theodoridis320.36
Priya Goyal474620.39
Zachary Devito520.36
William S. Moses6161.97
Sven Verdoolaege770646.15
Andrew Adams893653.55
Albert Cohen9100272.30