Name
Abstract | ||
|---|---|---|
This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $dgeq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d$, hidden layer widths at most $w,$ and arbitrary depth can approximate any continuous function of $d$ variables arbitrarily well. It turns out that this minimal width is exactly equal to $d+1.$ That is, if all the hidden layer widths are bounded by $d$, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions. On the other hand, we show that any continuous function on the $d$-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly $d+1.$ Our construction gives quantitative depth estimates for such an approximation. |
Year | Venue | Field |
|---|---|---|
2017 | arXiv: Machine Learning | Discrete mathematics,Continuous function,Mathematical optimization,Arbitrary-precision arithmetic,Unit cube,Artificial neural network,Expressive power,Mathematics,Bounded function |
DocType | Volume | Citations |
Journal | abs/1710.11278 | 13 |
PageRank | References | Authors |
0.72 | 14 | 2 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Boris Hanin | 1 | 47 | 4.04 |
| Mark Sellke | 2 | 14 | 2.11 |