Name
Title | ||
|---|---|---|
<italic>AdaNS</italic>: Adaptive Non-Uniform Sampling for Automated Design of Compact DNNs |
Abstract | ||
|---|---|---|
This paper introduces an adaptive sampling methodology for automated compression of Deep Neural Networks (DNNs) for accelerated inference on resource-constrained platforms. Modern DNN compression techniques comprise various hyperparameters that require per-layer customization. Our objective is to locate an optimal hyperparameter configuration that leads to lowest model complexity while adhering to a desired inference accuracy. We design a score function that evaluates the aforementioned optimality. The optimization problem is then formulated as searching for the maximizers of this score function. To this end, we devise a non-uniform adaptive sampler that aims at reconstructing the band-limited score function. We reduce the total number of required objective function evaluations by realizing a targeted sampler. We propose three adaptive sampling methodologies, i.e., AdaNS-Zoom, AdaNS-Genetic, and AdaNS-Gaussian, where new batches of samples are chosen based on the history of previous evaluations. Our algorithms start sampling from a uniform distribution over the entire search-space and iteratively adapt the sampling distribution to achieve highest density around the function maxima. This, in turn, allows for a low-error reconstruction of the objective function around its maximizers. Our extensive evaluations corroborate AdaNS effectiveness by outperforming existing rule-based and Reinforcement Learning methods in terms of DNN compression rate and/or inference accuracy. |
Year | DOI | Venue |
|---|---|---|
2020 | 10.1109/JSTSP.2020.2992384 | IEEE Journal of Selected Topics in Signal Processing |
Keywords | DocType | Volume |
Compact deep neural networks,adaptive sampling,automated neural network compression,hardware-aware DNN design,multi-objective optimization | Journal | 14 |
Issue | ISSN | Citations |
4 | 1932-4553 | 1 |
PageRank | References | Authors |
0.35 | 0 | 4 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Mojan Javaheripi | 1 | 18 | 5.83 |
| Mohammad Samragh | 2 | 38 | 7.01 |
| Tara Javidi | 3 | 806 | 78.83 |
| Farinaz Koushanfar | 4 | 3055 | 268.84 |