Name
Abstract | ||
|---|---|---|
We revisit the bias-variance tradeoff for neural networks in light of modern empirical findings. The traditional bias-variance tradeoff in machine learning suggests that as model complexity grows, variance increases. Classical bounds in statistical learning theory point to the number of parameters in a model as a measure of model complexity, which means the tradeoff would indicate that variance increases with the size of neural networks. However, we empirically find that variance due to training set sampling is roughly textit{constant} (with both width and depth) in practice. Variance caused by the non-convexity of the loss landscape is different. We find that it decreases with width and increases with depth, in our setting. We provide theoretical analysis, in a simplified setting inspired by linear models, that is consistent with our empirical findings for width. We view bias-variance as a useful lens to study generalization through and encourage further theoretical explanation from this perspective. |
Year | Venue | Field |
|---|---|---|
2018 | arXiv: Learning | Training set,Statistical learning theory,Mathematical optimization,Linear model,Bias–variance tradeoff,Sampling (statistics),Artificial neural network,Mathematics,Model complexity |
DocType | Volume | Citations |
Journal | abs/1810.08591 | 1 |
PageRank | References | Authors |
0.35 | 25 | 7 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Brady Neal | 1 | 1 | 0.69 |
| Sarthak Mittal | 2 | 1 | 0.69 |
| Aristide Baratin | 3 | 25 | 2.69 |
| Vinayak Tantia | 4 | 1 | 0.35 |
| Matthew Scicluna | 5 | 1 | 0.35 |
| Simon Lacoste-Julien | 6 | 1138 | 62.72 |
| Mitliagkas, Ioannis | 7 | 250 | 15.95 |