Name
Abstract | ||
|---|---|---|
Deep neural networks need to be compressed due to their high memory requirements and computational complexity. Numerous compression methods have been proposed to solve this issue, but we still do not fully understand how the compression error will impact the neural networks. We take inspiration from the rate distortion theory to propose a new distortion function which measures the gap between the Bayes risk of a classifier before and after the compression. Since this distortion is not tractable, we derive a theoretical closed-form approximation when the last fully connected layer of a deep neural network is compressed with a uniform quantizer. This approximation provides insight into the relationship between the accuracy loss and some key characteristics of the neural network. Numerical simulations show that the approximation is reasonably accurate. |
Year | DOI | Venue |
|---|---|---|
2021 | 10.1109/SSP49050.2021.9513733 | 2021 IEEE Statistical Signal Processing Workshop (SSP) |
Keywords | DocType | ISSN |
computational complexity,rate distortion theory,distortion function,closed-form approximation,fully connected layer,distortion approximation,compressed softmax layer,deep neural networks,Bayes risk | Conference | 2373-0803 |
ISBN | Citations | PageRank |
978-1-7281-5768-9 | 0 | 0.34 |
References | Authors | |
0 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Diana Resmerita | 1 | 0 | 0.34 |
| Rodrigo Cabral Farias | 2 | 0 | 0.34 |
| Benoît Dupont de Dinechin | 3 | 3 | 1.07 |
| Lionel Fillatre | 4 | 0 | 0.34 |